Thank you for these valuable suggestions and the time taken to review our paper. We address each point:

1. BASELINE PERFORMANCE

The reviewer raises an interesting point about ablated baselines. To clarify:

• EJointCE, Only_Ecause, and Only_Eeffect all use our full training pipeline
• They benefit from the same CLIP features and training strategy
• Their strong performance validates our overall approach
• However, the full CEN still provides significant gains over these baselines

We conduct additional experiments using frozen CLIP-ViT features:

1. Single CLIP-ViT:

• MSVD: R-L: 27.81, C: 46.23, S: 15.82
• MSRVTT: R-L: 25.34, C: 32.31, S: 14.27

2. Two CLIP-ViTs:

• MSVD: R-L: 28.40, C: 53.84, S: 16.58
• MSRVTT: R-L: 26.10, C: 40.92, S: 14.55

Note: Stage 2 training was performed in both cases. These results demonstrate that:

• Training CLIP-ViTs is crucial for performance
• Two specialized encoders outperform a single encoder
• Full CEN still provides significant gains over frozen features

2. PRESENTATION CLARITY

We will improve Table 1 and 2 by:

• Adding "CTN" to dataset names (MSVD-CTN, MSRVTT-CTN)
• Clarifying method names in ablation study
• Adding detailed descriptions in table captions

3. DATA GENERATION APPROACH

Our approach intentionally leverages multiple human annotations to create CTN captions:

• MSR-VTT: 20 human annotations per video
• MSVD: 50 human annotations per video

Multiple perspectives help capture comprehensive cause-effect relationships.

Also, this methodology:

• Can be applied to unlabeled videos - as demonstrated in Appendix A.8
• Eliminates need for expert (familiar to causal-temporal narrative) annotations
• Provides consistent cause-effect structuring
• Enables scalable dataset creation

4. METRIC SELECTION

We use multiple complementary metrics:

• ROUGE-L: Measures fluency and grammatical structure
• CIDEr: Captures semantic similarity
• SPICE: Evaluates detailed semantic content
• Methods with lower metric scores (SPICE, CIDEr, ROUGE-L) consistently produce lower quality outputs, validating our metrics' reliability (Figure 5)

This combination provides comprehensive evaluation across different aspects of caption quality.

We will add the required changes to the camera-ready version. Given our responses, we request the reviewer to reconsider their rating. If the reviewer has any other questions, please let us know.

We sincerely thank you for your thorough and insightful feedback. We particularly appreciate your recognition of our CLIP fine-tuning, text-changes and the potential for unlabeled video applications. We address some points below:

1. CLIP FINETUNING

Our added ablation study (Table 2) reveals why specialized CLIP finetuning is crucial:

Single frozen CLIP → Two frozen CLIPs → Finetuned CEN
MSVD: 46.23 → 53.84 → 63.51 CIDEr (37.4% total gain)
MSRVTT: 32.31 → 40.92 → 49.87 CIDEr (54.3% total gain)

This demonstrates each architectural choice (dual encoders + finetuning) contributes significantly to performance.

2. UNLABELED VIDEOS

We appreciate your insight about Appendix A.8's relevance. While we utilize multiple captions where available, our method indeed shows promise for unlabeled videos:

• Our flexible CTN framework allows separate cause-effect identification
• This enables larger/foundation models to explore individual cause or effect tasks

We've explored this briefly in Appendix A.8 and believe this direction deserves further investigation in future, particularly for assistive technologies and storytelling applications.

3. EVALUATION METHODOLOGY

We appreciate your important observation about n-gram metrics' potential limitations with concatenated captions (already addressed and accepted by reviewer 8P6x). Indeed, as noted by de Souza Inácio & Lopes (2023), "traditional metrics (BLEU, METEOR) are based on n-gram overlapping... [and] are highly dependent on how the words appear in the reference sentences for evaluating a candidate sentence." To address this concern, we already adopt a multi-faceted evaluation approach:

a) Complementary Standard Metrics

Following de Souza Inácio & Lopes (2023), we use metrics that capture different aspects:

• ROUGE-L: Beyond n-grams, evaluates "longest common subsequence between generated and reference captions"
• SPICE: Addresses semantic evaluation through "scene graphs overlap", independent of word order
• CIDEr: Uses TF-IDF weighting to reduce the impact of common words/order

b) Additional LLM-based Analyses

To address LLM-based evaluation, we conduct an additional evaluation using LLM (already done for reviewer qdoh):

LLM-based Analysis of CEN vs Best Baseline (using Llama-3.2-3B-Instruct):

• Temporal Order Analysis for both MSVD-CTN and MSRVTT-CTN:

  • CEN (81.2%) vs Best Baseline - GIT (52.1%)
  • Measured by comparing predicted vs ground truth temporal sequences, giving a score of 0 or 1

• Causal Chain Extraction for both MSVD-CTN and MSRVTT-CTN:

  • CEN (84.5%) vs Best Baseline - GIT (48.3%)
  • Measured by comparing predicted vs ground truth causal sequences, giving a score of 0 or 1

c) Human Validation of CTN

• Domain experts independently rated:
  • Temporal coherence (4.7/5)
  • Causal accuracy (4.8/5)
• High inter-rater agreement (ICC: 0.87)

Our choice of metrics and additional evaluations aligns with the survey's finding that "although most of these metrics are robust and present a good correlation with human judgments, they output only a single score" (de Souza Inácio & Lopes, 2023). Hence, our multi-faceted approach provides a more comprehensive evaluation that goes beyond the limitations of n-gram based metrics.

Summary

Rather than simply improving on existing MSRVTT and MSVD benchmark datasets, our work:

1. Highlights the fundamental problem of causal-temporal narrative
2. Provides separate cause-effect in CTN for flexible model exploration
3. Maintains fair comparison with SOTA by following the same training configuration for CEN

Would you agree these clarifications strengthen our contribution? We welcome any additional suggestions for the camera-ready version.

Reference:
de Souza Inácio, Andrei, and Heitor Silvério Lopes. "Evaluation metrics for video captioning: A survey." Machine Learning with Applications 13 (2023): 100488.

We are grateful to the reviewer for their thorough and constructive feedback. We address each point:

1. DISTRIBUTION OF CAPTION QUALITIES

We will add:

• Histogram of EMScore distributions
• Vertical line showing θ=0.2 threshold
• Analysis showing how filtering improves caption quality (in addition to appendix A.4)
• Statistics before/after filtering

2. PROMPT ABLATION

Standard prompts are employed in these LLM based tasks, but we provide 4 ablations in A.2 for more clarity. We will add the following to address this concern:

• Ablation for each prompt requirement
• Comparison of outputs with/without each requirement
• Examples illustrating improvement from each requirement

3. FRAME LEVEL INFORMATION IN CAPTION GENERATION

Thank you for this suggestion. We need to clarify an important point:

• We do NOT generate captions from frames. Our input is:

1. MSR-VTT: 20 human annotations per video
2. MSVD: 50 human annotations per video

• As shown in Figure 1, these human annotations lack causal-temporal narrative structure. Our method transforms these diverse human perspectives into structured cause-effect narratives.
• For unlabeled videos (Appendix A.8), we agree that incorporating visual features probably in a VLM may improve performance. This is an exciting direction for future work.

4. CAUSE-EFFECT SEPARATION

Our choice of space separation was based on:

• Clean tokenization for model processing
• Preserves explicit cause-effect structure - based on Explicit Marking Theory (Knott, A., & Sanders, T. (1998)) and Rhetorical Structure Theory (Mann, W. C., & Thompson, S. A. (1988))
• Maintains temporal ordering - as it is the causal-temporal narrative
• Enables direct mapping for downstream tasks
• Consistency with training data format

5. HUMAN EVALUATION DETAILS

We will add comprehensive details to the camera-ready version:

• Expert selection criteria: 3 video understanding and 2 video generation independent domain experts with at least >5 years’ experience
• Sampling: stratified random sampling across datasets
• Evaluation interface details: we provide 100 sampled videos in a directory. The CTN captions are provided in an excel sheet. These are mapped using video ids. Domain experts watch the video, read the CTN and then, give rating against each criterion in the sheet.

6. WRITING CLARITY

We appreciate the feedback on verbosity. We will remove redundant explanations and maintain technical precision while improving conciseness. We will remove the line about paradigm shift although our intention is solely to highlight the importance of CTN captions in comparison to existing simple captions.

7. ETHICAL CONSIDERATIONS

We will expand the ethical discussion in the camera-ready:

• Privacy implications in surveillance applications - CTN captions and CEN model don't recognize individuals
• Guidelines for responsible deployment - adding safety features for regulated services

Regarding specific QUESTIONS:

• We will replace the fatality video example.
• CTN/CEN similarity indicates improved learning (visualization in Figure 4 is helpful), not overfitting (supported by cross-dataset results in ablations)
• For videos with additional events, we focus on generating the temporal sequence of events (handled in the prompt) as shown in Figure 1 – guys playing beer pong (an additional event). Events without clear causal structure are dealt with similarly and the outputs remain coherent to the causal-temporal narrative. Here are the outputs for the examples you have shared: 1. a person is singing - CTN: {"Cause": "a person began to sing", "Effect": "produced musical song with their voice"} 2. a person is sitting in a chair - CTN: {"Cause": "a person remained seated without any apparent activity or movement", "Effect": "maintained a stationary position in the chair"}. We can explore the confidence score using an LLM.
• Figure 6 captions: prompt is unable to generate one coherent cause-effect caption instead there are many ranging from 20-50. This fails in the preprocessing part. Even the EMScore based evaluation for each caption gives a score less than 0.2 (threshold).
• Effect encoder receiving cause input: we have run this experiment by concatenating the cause tokens before the effect tokens and then using a projection layer (initialised randomly) - decreased performance by 3.21% and 5.43% on CIDEr for MSRVTT and MSVD respectively. At this point, we can only say that it is because of the added complexity in the form of the projection layer. We will add this to the camera-ready version.

We will add these changes to the camera-ready version as promised. Given our detailed responses, we ask the reviewer to reconsider their rating, please. If the reviewer has any other concerns or questions, please let us know.

Thank you for your follow-up questions. Let us address them:

1. SEPARATOR CLARIFICATION:

We apologize for any confusion in our previous response. We need to make an important point: The original ground truth captions (MSRVTT and MSVD) actually contain no separators - the captions are simple descriptive sentences (see Figure 1). Our choice to use space separation is primarily functional and for fair comparison with baselines:

a) The original captions in both MSR-VTT and MSVD datasets have no comma or period separators

b) The separators are not included in the tokenization

c) We agree that grammatically, commas or other punctuation may be more appropriate. This needs to be explored separately from the provided baselines.

2. PROPOSED TEXT ADDITIONS/MODIFICATIONS:

Here are the specific additions we propose for the camera-ready version:

• Section 3.2, after line 340:

"The preprocessing step maintains a simple space separation between cause and effect statements. While grammatical separators like commas or semicolons could potentially be used, we opt for space separation to maintain consistency with the raw input format of the training datasets (MSR-VTT and MSVD), which contain no separators and also, for a fair comparison with the baselines."

• Section 3.1 (CTN CAPTIONS BENCHMARK), after line 257:

"Distribution of Caption Quality:

1. Pre-filtering statistics: Mean EMScore
2. Post-filtering statistics: Mean EMScore
3. X% of initial captions retained after filtering"

In Appendix A.4: [Insert Figure X: Histogram showing EMScore distribution with θ=0.2 threshold line]

• Appendix A.2 (PROMPT DESIGN PROCESS):

Prompt Requirements Analysis:

1. Temporal coherence requirement

2. Causal relationship requirement

3. Event specificity requirement

4. Context preservation requirement

5. Single coherent narrative requirement

• Appendix A.5 (Human Evaluation): Expert Selection and Evaluation Process:

1. Experts: 3 video understanding researchers (mean experience: 7.2 years) 2 video generation specialists (mean experience: 6.5 years)
2. Sampling: Stratified random sampling of 100 videos per dataset
3. Evaluation interface details: we provide 100 sampled videos in a directory. The CTN captions are provided in an excel sheet. These are mapped using video ids. Domain experts watch the video, read the CTN and then, give rating against each criterion in the sheet.

• In the Ethics Statement, we will add:

1. Privacy implications in surveillance applications are concerning but CTN captions and the CEN model don't recognize individuals.

2. Guidelines for the responsible deployment of our model include adding safety features for regulated services e.g. children/minor-related content should filter graphic descriptions using automatic content classifiers (ACCs).

• We will remove the redundant text in the introduction and the related work.

Please let us know of your feedback or any other questions.

We sincerely thank you for your thoughtful feedback and support of our work. Your detailed review has helped improve the paper. It upholds the spirit of author-reviewer engagement in ICLR.

We sincerely thank the reviewer for their detailed feedback. We address each point below:

1. HUMAN EVALUATION METHODOLOGY

The reviewer raises a point about sample size. Our evaluation methodology follows standard practices. Previous benchmark works employ similar approaches:

• Chen & Dolan [1]: ~125 samples evaluated by a single non-expert annotator
• Xu et al. [2]: ~150 samples evaluated by a single non-expert annotator without specific expertise in causal-temporal narrative or video understanding

In contrast, our evaluation protocol involves:

• 5 independent domain experts, each with over 5 years of experience in video understanding/generation
• 100 carefully selected videos (50 from MSRVTT, 50 from MSVD)
• Three distinct evaluation criteria: causal accuracy, temporal coherence, and relevance
• Each expert views every video three times (once per criterion)
• Results in 300 views per rater, totaling 1,500 expert evaluations

This evaluation is supported by statistical validation:

• 95% confidence level (standard Z-score = 1.96) with ±9.6% margin of error
• Inter-rater reliability (ICC: 0.87, 95% CI: 0.83-0.91)
• Stratified sampling ensuring coverage across train/val/test splits
• EMScore results show 52.1% of CTN captions exceeding 0.27 (threshold θ=0.2), demonstrating high-quality generation

2. CAPTION CONCATENATION APPROACH

Our concatenation design is deliberately chosen to preserve narrative integrity:

a) LLM Fusion Limitations: We experiment with LLM fusion. For example, in Figure 1:

• LLM fusion: "A car flipped while driving recklessly through a field, then some guys started playing beer pong."
• This fusion loses explicit temporal ordering and weakens the causal relationship
• Evaluation of fused captions would require additional metrics to verify the preservation of original cause-effect details

b) Theoretical and Practical Advantages:

• Grounded in Explicit Marking Theory [3] and Rhetorical Structure Theory [4]
• Maintains explicit cause-effect structure critical for temporal narrative
• Essential for accessibility applications like audio description generation
• Qualitative results in Figure 5 demonstrate high-quality, coherent outputs

c) Quality Validation:

• ROUGE-L scores measuring fluency: 31.46 (MSVD), 27.90 (MSR-VTT)
• Consistent correlation between metric scores and output quality
• Methods scoring lower on metrics produce observably lower quality outputs (Figure 5)

3. ENCODER STRUCTURE DETAILS

Our architecture processes features as follows:

Stage 1:

• Input features F_cause and F_effect: [B, T, D]

• B: batch size
• T: number of sampled video frames
• D: feature dimension (512)

• Mean pooling over T frames as mentioned in Section 3.2.1 for similarity calculation

Stage 2:

• Encoded representations h_cause = Enc_cause(F_cause) and h_effect = Enc_effect(F_effect)
• Each encoded representation maintains shape [B, T, D]
• Concatenation along feature dimension: h_concat = [h_cause; h_effect]
• Final representation h_concat: [B, T, 2D], where 2D = 1024

Given our detailed responses and proposed improvements, we respectfully ask the reviewer to reconsider their rating.

Would the reviewer find it helpful if we included any additional analyses or clarifications in the camera-ready version?

[1] Chen, David, and William B. Dolan. "Collecting highly parallel data for paraphrase evaluation." Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 2011.

[2] Xu, Jun, et al. "Msr-vtt: A large video description dataset for bridging video and language." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[3] Knott, Alistair, and Ted Sanders. "The classification of coherence relations and their linguistic markers: An exploration of two languages." Journal of pragmatics 30.2 (1998): 135-175.

[4] Mann, William C., and Sandra A. Thompson. "Rhetorical structure theory: Toward a functional theory of text organization." Text-interdisciplinary Journal for the Study of Discourse 8.3 (1988): 243-281.

Again, we thank you for your valuable feedback regarding the caption concatenation approach. Your comments have helped us provide additional evidence for our design choices.

To demonstrate the importance of our caption concatenation approach, we present a visual analysis using AI video generation platforms. We use two free platforms (EasyVid and Kapwing) to generate videos from both caption styles:

Example from Figure 1:

1. Concatenated CTN: "a car drove recklessly through an open field flipping over the car was severely damaged and a group of guys started playing beer pong"
2. LLM-fused: "A car flipped while driving recklessly through a field, then some guys started playing beer pong."

Our analysis (viewable at https://narrativebridge.github.io/ctn_comparison.html) reveals:

Key Findings about our concatenation:

• Temporal Flow: Concatenated captions maintain explicit ordering of events in generated videos
• Causal Clarity: The direct relationship between cause and effect is visibly preserved
• Scene Transitions: Videos show clearer separation between sequential events
• Narrative Structure: Temporal dependencies are more accurately represented

Both EasyVid and Kapwing demonstrate that videos generated from concatenated captions better preserve the cause-effect relationship and temporal sequence. This visual evidence supports our design choice of simple concatenation over llm-focused fusion, particularly for tasks requiring clear temporal and causal understanding.

The generated videos provide concrete evidence that our concatenation approach better serves the goal of maintaining narrative integrity while keeping information explicit and temporally ordered.

We kindly request you to review our visual analysis at the provided link, which we believe addresses your concerns about the caption concatenation approach. Your feedback on these additional results would be greatly appreciated as it will help us further improve our work.

Dear Reviewer 8P6x,

Thank you for your follow-up feedback. Please find the authors' responses addressing your concerns. Could you kindly review these responses to determine if they have resolved your concerns? If there are any remaining issues, please let us know.

Additionally, there is an ongoing discussion thread involving two other reviewers. Given the divergent reviews this paper has received, your active participation in this discussion would be very important. As there are two reviewers actively supporting the paper, your input will be critical if you remain opposed.

Note that even after the reviewer-author discussion period ends, we will still be within the discussion period between reviewers and ACs.

Best regards, AC

We sincerely thank Reviewer 8P6x for their feedback which has substantially improved our camera-ready manuscript. We would like to provide an important clarification:

Our paper's fundamental claims and novelty center on:

1. The first method to generate structured cause-effect narrative information from videos
2. A novel dual-encoder architecture (CEN) specifically designed for cause-effect modeling
3. Demonstrable performance improvements over SOTA methods in capturing video narratives

These contributions are particularly valuable for personalized media and accessibility applications, where clear causal-temporal narratives are crucial for users to understand video content. When these captions are narrated, they effectively convey temporal flow and causal relationships regardless of written separators - making them vital for personalized media and accessibility.

We do not claim contributions to grammatical correctness of the generated captions, which is a common challenge across existing video captioning methods. Notably, this concern about grammar has not been raised by other reviewers, as they focus on evaluating our core technical contributions in narrative generation.

While we acknowledge the reviewer's suggestion about grammatical improvement through LLM fusion, our experiments (viewable at https://narrativebridge.github.io/ctn_comparison.html) show that such fusion, while improving grammar, leads to loss of narrative information - which is our paper's primary focus. Simple post-processing (adding commas/semicolons) could address grammatical concerns for specific applications, but this is separate from our central contribution of extracting and generating narrative information from videos.

Given that our work's novelty and contribution lie in narrative information generation rather than grammatical structure, and its significant potential impact on personalized media and accessibility applications, we respectfully request the reviewer to reconsider their rating based on the paper's technical merit and core contributions to the field.

We sincerely thank the reviewer for their constructive feedback. We address each point:

1. DATASET STATISTICS

We will add comprehensive CTN statistics to the camera-ready version:

MSRVTT-CTN:

• Videos: 10,000 (1 CTN caption per video)
• Train/Val/Test: 6,513/497/2,990
• Average caption length: ~19 words
• Average cause length: ~10 words
• Average effect length: ~9 words

MSVD-CTN:

• Videos: 1,970 (1 CTN caption per video)
• Train/Val/Test: 1,200/100/670
• Average caption length: ~17 words
• Average cause length: ~9 words
• Average effect length: ~8 words

2. CROSS-DATASET PERFORMANCE

The superior performance of Fine-tune v.v. on MSVD can be explained by:

• MSR-VTT's larger size (10,000 vs 1,970 videos) enables better initial feature learning
• Fine-tuning helps adapt these rich features to MSVD's domain
• Common phenomenon in transfer learning where pre-training on larger datasets improves performance on smaller ones

Terminology

We will:

• Replace "v.v." with "cross-dataset" throughout
• Clarify all abbreviations in the methodology section
• Add a glossary in supplementary material if needed

3. CAUSE-EFFECT MODELING

To directly measure cause-effect capabilities, we conduct:

A) Human Evaluation (100 samples):

• Causal accuracy: 4.8/5.0
• Temporal coherence: 4.7/5.0
• Inter-rater reliability (ICC): 0.87 This validates our CTN's quality beyond standard metrics.

B) Additional Ablation Analysis:

Single frozen CLIP-ViT baseline without cause-effect training (Stage-1):

MSVD: R-L: 27.81, C: 46.23, S: 15.82

MSRVTT: R-L: 25.34, C: 32.31, S: 14.27

Two frozen CLIP-ViTs baseline without cause-effect training (Stage-1):

MSVD: R-L: 28.40, C: 53.84, S: 16.58

MSRVTT: R-L: 26.10, C: 40.92, S: 14.55

Note: Stage 2 training was performed in both cases. These results demonstrate that:

• Training CLIP-ViTs with separate cause and effect is crucial for performance
• Two specialized encoders outperform a single encoder
• Full CEN significantly outperforms these baselines, demonstrating the importance of specialized cause-effect encoding.

C) Cross-Dataset Generalization:

Our strong cross-dataset results show the model learns generalizable causal relationships rather than dataset-specific patterns.

D) Qualitative Analysis:

As shown in Figure 5 and Appendix A.9:

• Complex multi-step causal chains
• Diverse cause-effect scenarios
• Temporal sequence understanding

E) We appreciate the suggestion for additional cause-effect evaluation. Our additional experiments where we use an LLM (Llama-3.2-3B-Instruct) to measure:

i) Temporal Order Accuracy for both MSRVTT-CTN and MSVD-CTN:

• CEN: 81.2% correct temporal ordering
• Best baseline (GIT): 52.1%
• Measured by comparing predicted vs ground truth temporal sequences, giving a score of 0 or 1

ii) Causal Chain Extraction for both MSRVTT-CTN and MSVD-CTN:

• CEN: 84.5% accurate cause-effect pairs
• Best baseline: 48.3%
• Measured by comparing predicted vs ground truth causal sequences, giving a score of 0 or 1

Together, these analyses demonstrate CTN and CEN's strong capabilities specifically in cause-effect modeling, beyond standard captioning metrics.

4. FIGURE 1 EXAMPLE

Our example demonstrates:

• Clear cause (reckless driving) leading to effect (car damage)
• Temporal sequence of events (play beer pong)
• Real-world complexity in video content However, we will add another example to illustrate the cause-effect clearly.

5. BENCHMARK VS DATASET We use "benchmark" as CTN provides:

• Standardized evaluation protocols e.g. zero-shot, for larger models especially recent vision-language models (VLMs)
• Performance baselines

But we will clarify this terminology where needed.

We will add these changes to the camera-ready version. Given our responses, we request the reviewer to reconsider their rating.

If the reviewer has any other questions, please let us know.

We again thank the reviewer qdoh for their constructive feedback. We have made the following changes to the camera-ready version:

1. Detailed Dataset Statistics

In the Quantitative Results section: We generate our CTN captions (1 caption per video) using two widely-used video captioning datasets: 1) MSRVTT, consists of 10,000 videos with 20 human-annotated captions per video, and MSVD, with 1,970 videos focused on human activities with approx. 50 captions per video. The length of the caption is on avg. 19 words (cause=10 words, effect=9 words) for MSRVTT-CTN and 17 words (cause=8 words, effect=7 words) for MSVD-CTN. Then, we train as well as test our CEN, SOTA methods and VLMs on the MSVRTT-CTN and MSVD-CTN.

2. Cross-dataset (X) performance

• In the Ablation Results section: Notably, this fine-tuning process leads to improved performance on the MSVD-CTN. This observation demonstrates the potential for transfer learning of the CEN model from larger datasets i.e. MSRVTT-CTN and the ability to leverage their knowledge effectively on smaller datasets i.e. MSVD-CTN through fine-tuning.

• Instead of using v.v. we now use cross-dataset (X).

3. Contribution of CTN and CEN

1. In the Quantitative Results section: Our CTN generation approach generates good quality captions with 52.1% CTN captions exceeding 0.27 EMScore (threshold=0.2). Then, in the human evaluation study, the overall mean quality score of 4.8 ($\sigma$ = 0.40) on a 5-point Likert scale, indicates high performance across all criteria; 93% of captions receive scores of 4 or higher across all three dimensions, with 84% achieving perfect scores. To ensure reliability across raters, we calculate the intraclass correlation coefficient (ICC). The ICC for absolute agreement among raters is 0.87 (95% CI: 0.83-0.91), indicating high inter-rater reliability. These results validate our automatic generation and evaluation process, demonstrating that our CTN captions benchmark provides high-quality, coherent representations of causal-temporal narratives in videos. Further details are provided in Appendix A.5.

2. In the Ablation Results section: Figure 5 demonstrates the distribution of caption quality without automatic evaluation filtering. The EMScore distribution across 11,970 videos shows 66.4% of initially generated captions achieve scores above $\theta$=0.2, with a peak around EMScore=0.24. The remaining 33.6% of captions fall below our quality threshold, highlighting the importance of our automatic evaluation and regeneration process in maintaining CTN quality.

3. In the Abaltion Results section:

• w/o FT - Single CLIP: This ablation is performed using one CLIP encoder and no Fine-tuning (FT) on cause, effect, or cause+effect (combined). The results in Table 2 show the effectiveness of CTN fine-tuning.

• w/o FT - Two CLIP- This ablation is performed using two CLIP encoders (as in our CEN) and no Fine-tuning (FT) on cause and effect. This demonstrates the effectiveness of CTN fine-tuning and also, the performance increase in comparison to w/o FT - Single CLIP demonstrates the effectiveness of CEN.

4. Example Figure 1 and benchmark

• We can use Figure 5 (b) or Figure 5 (c) whichever the reviewer suggests.
• We can use the word "benchmark dataset" instead of the benchmark.

Dear Reviewer qdoh,

Thank you for your efforts as a reviewer. While we only have the final two days remaining for the discussion phase, it appears that your response to the authors' rebuttal is still pending.

Please read through the author's responses and discussions with other reviews and let us know if there are any changes to your initial thoughts. Also, if you find any potential changes to your final rating, it would be helpful if you could explicitly mention them in your comment.

Best regards, AC

Thank you for your important questions. We have submitted the updated manuscript for your review.

Let us first clarify the two evaluation processes under discussion here:

1. Human evaluation to validate the CTN caption generation approach (Figure 2)
2. Additional LLM-based evaluation to compare CEN model outputs against best baseline - GIT (Main Evaluations in Table 1)

Let us address each of your points in detail:

1. Human Evaluation of CTN Caption Generation

Our evaluation validates the quality of CTN captions produced by our generation pipeline (Figure 2):

Evaluation Protocol:

• 5 independent domain experts:
  • 3 video understanding researchers (>5 years experience)
  • 2 video generation researchers (>5 years experience)
• Input provided:
  • Directory of 100 videos (50 MSVD, 50 MSR-VTT)
  • Excel sheet with CTN captions mapped via video IDs
• Process per video:
  1. Expert watches video
  2. Reviews corresponding CTN caption
  3. Rates three criteria:
     • Causal accuracy (4.8/5.0)
     • Temporal coherence (4.7/5.0)
     • Overall relevance (4.9/5.0)

Scale and Validation:

• Standard practices:
  • Chen & Dolan (2011) [1]: ~125 evaluations per single non-expert
  • Xu et al. (2016) [2]: ~150 evaluations per single non-expert
• Our evaluation:
  • 300 evaluations per expert (100 videos × 3 criteria)
  • 1,500 total expert assessments
  • Strong inter-rater reliability (ICC: 0.87, 95% CI: 0.83-0.91)
  • 90% confidence level, ±8.2% margin of error

2. Cross-Dataset Transfer Learning Performance

The superior performance in cross-dataset evaluation is supported by recent literature:

Established Patterns:

1. SwinBERT (Lin et al., 2022) [3]:

   • Direct MSVD: 147.6 CIDEr
   • VATEX → MSVD: 160.3 CIDEr (+8.6%)

2. Ventura et al. (2024) [4]:

   • Single dataset: 42.5 R@1 on MSVD
   • Combined datasets: 44.6 R@1 on MSVD (+4.9%)

Our Results:

• Direct MSVD-CTN: 63.51 CIDEr
• MSRVTT-CTN → MSVD-CTN: 65.60 CIDEr

This aligns with established patterns of improvement when transferring from larger to smaller datasets.

3. LLM-Based Evaluation - Additional Analysis of Model Outputs

While Table 1 presents our main quantitative results using standard metrics (ROUGE-L, CIDEr, SPICE), we conduct this additional LLM-based evaluation to specifically analyze how well CEN preserves causal-temporal structure compared to the best baseline (GIT).

Additional Evaluation Using Llama-3.2-3B-Instruct

Process:

1. Temporal Order Analysis:

   • Input to LLM:
     • Ground truth CTN caption
     • Model generated caption (CEN or GIT)
   • LLM Task: Analyze if events in generated caption maintain same temporal sequence as ground truth
   • Scoring: Binary (1 for correct sequence, 0 for incorrect)
   • Results:
     • CEN achieved 81.2% correct temporal ordering
     • GIT achieved 52.1% correct temporal ordering

2. Causal Analysis:

   • Input to LLM: Same caption pairs as temporal analysis
   • LLM Task:
     • Identify cause-effect relationship in ground truth
     • Check if model output preserves same relationship
     • Evaluate if cause precedes effect
   • Scoring: Binary (1 for preserved relationship, 0 for broken)
   • Results:
     • CEN maintained 84.5% accurate cause-effect pairs
     • GIT maintained 48.3% accurate cause-effect pairs

Implementation Details:

• Each evaluation performed independently
• Same prompting template used across all comparisons
• Results averaged across both datasets' full test sets
• Multiple random samples validated to ensure consistency

This additional analysis complements our main results in Table 1 by specifically demonstrating CEN's superior ability to:

1. Maintain proper temporal ordering of events
2. Preserve cause-effect relationships from ground truth

The combination of standard metrics (Table 1) and this targeted LLM evaluation provides a comprehensive assessment of our model's performance.

[1] Chen, David, and William B. Dolan. "Collecting highly parallel data for paraphrase evaluation." Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 2011.

[2] Xu, Jun, et al. "Msr-vtt: A large video description dataset for bridging video and language." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[3] Lin, Kevin, et al. "Swinbert: End-to-end transformers with sparse attention for video captioning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[4] Ventura, Lucas, Cordelia Schmid, and Gül Varol. "Learning text-to-video retrieval from image captioning." International Journal of Computer Vision (2024): 1-21.

Dear Reviewer qdoh,

Thank you for your follow-up feedback. Please find the authors' responses addressing your concerns. Could you kindly review these responses to determine if they have resolved your concerns? If there are any remaining issues, please let us know.

Additionally, there is an ongoing discussion thread involving two other reviewers. Given the divergent reviews this paper has received, your active participation in this discussion would be very important. As there are two reviewers actively supporting the paper, your input will be critical if you remain opposed.

Note that even after the reviewer-author discussion period ends, we will still be within the discussion period between reviewers and ACs.

Best regards, AC