Accident Anticipation via Temporal Occurrence Prediction

Dear Reviewer o9V3,

Thank you for your detailed review and valuable feedback on our submission "Accident Anticipation via Temporal Occurrence Prediction". We appreciate your recognition of our method's strong motivation, its effective comparison with conventional approaches, and the overall clarity of our paper's presentation.

We have carefully considered your comments, especially regarding the originality claims, missing citations, and the need for clearer ablation studies, and we're committed to addressing them to improve our manuscript.

Response to Weaknesses and Questions:

W1. Clarity on Our Claims: We sincerely apologize for any ambiguity or confusion caused by our claims. We have provided clarifications for each of our claims and guarantee that these will be revised in the accepted version of the paper to prevent any future misunderstanding for readers.

• "Existing methods typically predict frame-level anomaly scores as risk indicators," in l.2-3: We would like to clarify that our contribution does not lie in the use of multi-frame input, but in predicting multiple timestamp risk sequence. We fully acknowledge that many existing methods, including [Kumamoto+, WACV’25], utilize multi-frame inputs and powerful temporal encoders such as Transformers to extract temporal features. However, our claim means that prior works predict a single anomaly score for each frame, while our method predicts a future sequence of anomaly scores for each frame, such as the likelihood of an accident occurring at +0.1s, +0.2s, ..., up to +2.0s. This allows our model to explicitly capture the temporal evolution of accident risk over a future horizon.

• "For example, a branch of works [3, 4, 5, 6, 7] regards the risks as 1 for frames within a fixed interval (e.g. 2.0s) before the annotated accident timestamp and 0 for frames sampled from safe driving scenarios," in l.26-28: We would like to clarify that while the paper [Suzuki+, CVPR18] and following papers do introduce adaptive loss functions—typically by applying penalty weights that decay with temporal distance from the accident frame—they still assign a binary label of 1 to all frames within a fixed interval (e.g., 2.0s before the accident). In other words, although the adaptive loss functions reduce the penalty for frames farther from the accident, the model is still optimized toward predicting a value close to 1 for all positive frames, which may not reflect the actual uncertainty or gradual buildup of risk over time. In contrast, we argue that only the accident timestamp itself can be precisely labeled, while risk at earlier frames is uncertain and should be predicted in a temporally resolved and probabilistic manner, rather than assigned a flat label. Our method addresses this by predicting a distribution of future accident probabilities, which better captures the evolving nature of risk.

W2. Lacking Important Citations: Thank you for bringing these important missing citations to our attention. We agree that including relevant prior work is crucial for paper quality and proper contextualization. We will revise our manuscript to include comprehensive discussions of: * [Kumamoto+, WACV25]: We will discuss their use of Transformer models and driver attention for accident anticipation, * [Suzuki+, CVPR18]: We will include their work on adaptive loss functions and large-scale incident databases. * [Karim+, TITS22]: We will discuss their Dynamic Spatial-Temporal Attention Network, acknowledging its contribution to early anticipation and its approach to spatio-temporal modeling. * Other works: We will investigate the latest work and incorporate it into our related work for discussion.

W3. Novelty and the Causes of Performance Improvement:

• Novelty: The key novelty of our work lies in the novel paradigm of predicting the temporal occurrence of an accident. Specifically, we move from predicting a single risk score for each frame to predicting a temporal sequence of future accident probabilities for each frame. This formulation allows the model to better capture the evolving nature of risk in dynamic driving scenes.
• Causes of Performance Improvement: Our method does not contain object detection modules, nor does it depend on complex architectural modifications. Since two of our contributions are the novel paradigm of predicting the temporal occurrence of an accident and the snippet encoder, we have highlighted their effectiveness in our ablation studies (see Sec. 5.3).

As shown in Table 4, our main contribution, "predicting the temporal occurrence" (TOP), brings performance improvement to mAUC$^{0.1}$ (0.0201) and mTTA$^{0.1}$ (0.0510), especially to AUC$_{0.0s}^{0.1}$ (0.1715).

Our second contribution, "snippet encoder" (SE), also brings performance improvement to mAUC$^{0.1}$ (0.1378) and mTTA$^{0.1}$ (0.3006), especially to AUC$_{0.0s}^{0.1}$ (0.2090), while experiments I, II, V, VI utilized RNNs to aggregate temporal features.

Based on the above, these ablation experiments demonstrate that our proposed novel contributions are effective and can lead to performance improvements.

Table 4: Ablation study on the CAP-DATA [8] dataset, where TOP denotes the temporal occurrence prediction, SE denotes the snippet encoder, and AAA denotes the anomaly appearance annotations.

Questions: Clarifying Performance Improvement Causes: Since we have clarified that our contributions could cause performance improvement in the above "W3" with ablation studies in Table 4, we will clarify the difference in implementation details between the conventional methods and ours in this response.

We list all the components used in previous works (including text input, driver attention, object detector, spatial encoder, GCNs, RNNs) in the following Table. Compared with previous works using a spatial encoder and RNNs to understand spatial and temporal features, we capture spatial and temporal features simultaneously by a snippet encoder, SlowOnly (a very vanilla backbone in video recognition). Additionally, we did not use any other structures (like object detectors and GCNs) or auxiliary information (like text input and driver attention) to ensure simplicity of implementation. Despite this simplicity, we also achieve SOTA performance (see Rebuttal 2. for Reviewer frVz) within fair comparison and Kaggle competition. This demonstrates that the improvement is due to the proposed method itself rather than novel detection methods and architectural changes.

[Bao+, ICCV21] W. Bao et al., “Drive: Deep reinforced accident anticipation with visual explanation”, in ICCV, 2021.

[Fang+, ITSM24] J. Fang et al., “Cognitive accident prediction in driving scenes: A multimodality benchmark”, in ITSM, 2024.

[Karim+, TITS22] M. M. Karim et al., “A dynamic spatial-temporal attention network for early anticipation of traffic accidents”, in TITS, 2022.

[Wang+, TIV23] T. Wang et al., “GSC: A graph and spatio-temporal continuity based framework for accident anticipation”, in TIV, 2023.

Dear Reviewer o9V3,

We hope that we have addressed your questions in our rebuttal. Do you have any further questions? We would be delighted to have a discussion.

Dear Reviewer nKX6,

Thank you for your valuable time and insightful feedback on our submission "Accident Anticipation via Temporal Occurrence Prediction". We appreciate your recognition of our novel accident anticipation paradigm and the introduction of a new evaluation protocol that accounts for false alarm rates. Your comments are crucial for improving the clarity and rigor of our work.

We have carefully considered each of your points and would like to address them as follows:

Response to Questions:

1. Distinction Between "Score" and "Probability": Thank you for pointing out this important issue. We would like to clarify that in our paper, “anomaly score” and “probability” refer to the same concept—a quantitative value indicating the likelihood of an accident. We will revise the manuscript to standardize the terminology for clarity.

• Clarification of the main contribution: The key distinction we aim to highlight is not the difference between “score” and “probability,” but rather the shift from predicting a single score/probability to predicting a temporal sequence of such scores/ probabilities. Specifically, previous methods predict one anomaly score/risk probability per frame, indicating the general likelihood of an accident. In contrast, for each frame, our method predicts a sequence of future probabilities, such as the likelihood of an accident occurring at +0.1s, +0.2s, ..., up to +2.0s. This allows our model to explicitly capture the temporal evolution of accident risk over a future horizon.

We appreciate the reviewer’s observation and will ensure that this distinction is clearly articulated in the revised manuscript.

2. Reporting False Alarm Rates (FAR) of Existing Methods: Thank you for the valuable comment. We agree that quantitative evidence is essential to support our claim regarding the limitations of existing methods under high false alarm rates (FAR). In practice, the false alarm rate is threshold-dependent, making direct comparisons between methods at a fixed threshold inappropriate. Consequently, to ensure a fair comparison, we follow a consistent evaluation protocol: we report the FAR of each method when it achieves 80% early warning recall rate. The results of this evaluation are summarized in Table 1.

Table 1: False Alarm Rate (FAR) of each method when achieving 80% early warning recall.

[1] W. Bao et al., “Drive: Deep reinforced accident anticipation with visual explanation”, in ICCV, 2021.

[2] J. Fang et al., “Cognitive accident prediction in driving scenes: A multimodality benchmark”, in ITSM, 2024.

[3] M. M. Karim et al., “A dynamic spatial-temporal attention network for early anticipation of traffic accidents”, in TITS, 2022.

[4] T. Wang et al., “GSC: A graph and spatio-temporal continuity based framework for accident anticipation”, in TIV, 2023.

Our method ("Ours") achieves a remarkably low False Alarm Rate of 0.01. This is significantly lower than all other compared methods: Drive [1] (0.59), CAP [2] (0.47), DSTA [3] (0.23), GSC [4] (0.19), and the Baseline (0.25). This result strongly demonstrates our method's superior capability in controlling false alarms. In real-world applications, a low False Alarm Rate is crucial because it indicates that the system can more accurately identify genuine accident risks without frequently issuing erroneous alerts. An excessively high false alarm rate (like 0.59 and 0.47) can lead to user fatigue and distrust, thereby diminishing the system's practical utility. That's why we proposed a new evaluation protocol that explicitly measures recall rate and Time-to-Accident (TTA) only under acceptable false alarm rates. We will include this evaluation and the corresponding results in the revised manuscript to substantiate our motivation and improve clarity.

3. Temporal Exponential Decay for Penalty Weights of Loss Function: Thank you for your thoughtful question regarding the temporal weights of our loss function, especially considering the idea that earlier anticipations might warrant lower penalties. Such mechanisms (e.g., [1]'s implementation) give a lower penalty weight to the loss for earlier anticipations of accident videos in a temporal exponential decay manner ($e^{-\max(0,y-t)}$), while the labels remain 1 for all frames in the accident video, as shown in Equation (12).

$L_p(\{a_t\}) = \sum_t -e^{-\max(0,y-t)} \log(a_t^0), \quad (12)$

Similarly, we can apply this mechanism (Exponential Weights) to our method. We conducted ablation studies of this mechanism on the Baseline method (in the way of predicting an anomaly score) and our method (in the way of predicting a sequence of probabilities at multiple timestamps in the future), as shown in the following table. This mechanism could bring a large improvement to the Baseline method of late anticipations (e.g., AUC$^{0.1}_{0.5s}$). However, it only brings a subtle improvement to our method. This is because we model the supervision signal of different timestamps before the accident independently (see Rebuttal 2 for Reviewer TYgU), while previous works model them as the same "anomaly score" (the uncorrected label of early anticipations may degrade the ability of late anticipations). Therefore, this experimental result proves that our method is more robust to the penalty weights of the loss function.

Table 2. Impact of Exponential Temporal Weights on the baseline and our method.

4. Code and Pretrained Model Release: Yes, we plan to release our code, pretrained models, and detailed documentation upon acceptance to support full reproducibility. Our method achieves state-of-the-art (SOTA) performance on both the MM-AU and Nexar datasets. More importantly, we propose a more reasonable and standardized evaluation protocol for accident anticipation. By releasing our implementation along with strong SOTA baselines, we aim to facilitate fair comparison and accelerate progress in future research on this task.

Dear Reviewer TYgU,

Thank you for your very positive and constructive review of our submission "Accident Anticipation via Temporal Occurrence Prediction". We greatly appreciate your recognition of our paper's clarity, the relevance and impact of our topic, and the well-structured experiments. Your confidence in our work is truly encouraging.

We acknowledge your valuable feedback regarding the Related Work section and the need for a more explicit discussion on how our method formalizes "gradual risk evolution." We agree that addressing these points will significantly strengthen our manuscript, clarify our contributions, and provide a more comprehensive context for readers.

We have carefully considered your questions and would like to address them as follows:

Response to Weaknesses and Questions:

1. Comprehensive Discussion of Related Works: Thank you for your valuable comments. In response, we have carefully surveyed and reviewed relevant prior works. We will make sure to include all of them explicitly in the updated manuscript to provide a more comprehensive contextualization of our contribution.

• Works using recall and Time-To-Accident (TTA) as evaluation metrics: To the best of our knowledge, all accident anticipation works use recall (AUC or AP) and Time-To-Accident (TTA) (or Time-To-Collision (TTC)) as evaluation metrics. For instance, DSA [1] first proposed a dataset and baseline for accident anticipation using AP and TTA. AdaLEA [2] proposed an adaptive loss for early anticipation using AP and TTC. DSTA [3] proposed to utilize dynamic temporal attention and GRU using AP and TTA. DADA [4] proposed to utilize driver attention for anticipation using AUC. CAP [5] proposed a multimodality benchmark for anticipation using AUC, AP, and TTA. Graph(Graph) [6] proposed a nested graph-based framework for early accident anticipation using AP and TTA.

[1] F.-H. Chan et al., “Anticipating accidents in dashcam videos”, in ACCV, 2017.

[2] T. Suzuki et al., “Anticipating traffic accidents with adaptive loss and large-scale incident DB”, in CVPR, 2018.

[3] M. M. Karim et al., “A dynamic spatial-temporal attention network for early anticipation of traffic accidents”, in TITS, 2022.

[4] J. Fang et al., “Dada: Driver attention prediction in driving accident scenarios”, in TITS, 2021.

[5] J. Fang et al., “Cognitive accident prediction in driving scenes: A multimodality benchmark”, in ITSM, 2024.

[6] N. Thakur et al., “Graph(Graph): A nested graph-based framework for early accident anticipation”, in WACV, 2024.

• Works modeling driving risk as a temporally evolving or probabilistic process: UString [7] proposed an uncertainty-based accident anticipation model that employs spatio-temporal relational learning. It sequentially predicts traffic accident probabilities by leveraging Bayesian neural networks to address the intrinsic variability of latent relational representations. DRIVE [8] regarded risk prediction as a Markov decision process and proposed a deep reinforcement learning approach with visual explanation. AccNet [9] formulated the risk by monocular depth-enhanced 3D modeling. CRASH [10] formulated the risk with object detection and context modeling. A branch of work [1, 2, 3, 5, 6, 7, 8] regarded risk evolving as a sequence over time and used RNNs to predict risk. However, the risks before the accident lack accurate labels for training.

[7] W. Bao et al., “Uncertainty-based traffic accident anticipation with spatio-temporal relational learning”, in ACMMM, 2020.

[8] W. Bao et al., “Drive: Deep reinforced accident anticipation with visual explanation”, in ICCV, 2021.

[9] H. Liao et al., “Real-time accident anticipation for autonomous driving through monocular depth-enhanced 3D modeling,” in AA&P, 2024.

[10] H. Liao et al., “CRASH: Crash recognition and anticipation system harnessing with context-aware and temporal focus attentions,” in ACMMM, 2024.

2. Formalizing and Measuring "Gradual Risk Evolution": Thank you for the question. We formalize the "gradual evolution" of driving risk as a sequence of future accident probabilities over time. Specifically, for each timestamp, our method predicts a probability sequence at each future timestamp, rather than a single anomaly score. This allows us to model how the likelihood of an accident changes and increases as the vehicle approaches the accident time. We will revise the Method section to clearly explain this issue.

As illustrated in Table 1, the label of the probability sequence evolves from 2.0s before the accident to 0.1s before the accident, like a one-hot vector iteratively shifting left. That's how we formalize the "gradual evolution" of driving risk, while a single anomaly score cannot formalize this evolution.

Table 1: Labels of the probability sequence at every timestamp within a future period (0.1s-2.0s), where the accident timestamp is 0.0s.

Dear Reviewer frVz,

Thank you for your thorough review and constructive feedback on our submission "Accident Anticipation via Temporal Occurrence Prediction". We highly appreciate your recognition of the significance of our research direction, the clarity of our paper, and the novelty of our proposed paradigm and evaluation protocol. Your insights are invaluable in helping us improve the quality of our work.

We have carefully considered each of your points and would like to address them as follows:

Response to Weaknesses and Questions:

1. Generalization to Diverse Environments and Sensor Configurations:

We agree with your valid point that discussing the generalization capabilities of our method is crucial. Here are our points:

• MM-AU Includes Diverse Environments: As stated in [1], MM-AU is a large-scale ego-view traffic accident dataset collected from public accident datasets (CCD, A3D, DoTA, DADA-2000) and various video stream sites (YouTube, Bilibili, Tencent), including diverse weather conditions (sunny, rainy, snowy, foggy), lighting conditions (day, night), scenes (highway, tunnel, mountain, urban, rural), and roadways (arterial, curve, intersection, T-junction, ramp). That's why we chose to train our method on it (unlike previous works primarily use CCD and A3D). Besides, experimental results on the validation subset of MM-AU achieved state-of-the-art, further demonstrating its generalizability across diverse environments.
• Nexar Dataset: We further test our method on the Nexar dataset released by Nexar Inc.. The dataset includes 1,500 annotated video clips of real-world traffic scenarios, encompassing different environmental conditions (lighting, weather) and scene types (urban, rural, highway, etc.). We compare our method with solutions of participants in the Kaggle competition "Nexar Dashcam Collision Prediction Challenge" hosted by Nexar Inc. As shown in Table 2 below, our method outperforms the Champion solution on both public and private leaderboards.
• Sensor Types: Although our current experiments use dashcam data, the snippet encoder could be replaced with other encoders to accommodate input from different sensor modalities (e.g., LiDAR, radar). However, the lack of annotated traffic accident datasets of other sensor modalities prevents us from validating our method's performance with other sensor modalities.

[1] J. Fang et al., “Abductive ego-view accident video understanding for safe driving perception”, in CVPR, 2024.

2. Baseline Selection and State-of-the-Art Comparison:

• More Baselines: We further compared our method with two recent SOTA open-source baselines. We trained their model with the same implementation details on MM-AU (CAP-DATA) and demonstrate the experimental results in the table below. The quantitative results in the table demonstrate that our method outperforms recent SOTA methods.

Table 1: Quantitative results comparison of different methods on the CAP-DATA dataset.

[2] W. Bao et al., “Drive: Deep reinforced accident anticipation with visual explanation”, in ICCV, 2021.

[3] J. Fang et al., “Cognitive accident prediction in driving scenes: A multimodality benchmark”, in ITSM, 2024.

[4] M. M. Karim et al., “A dynamic spatial-temporal attention network for early anticipation of traffic accidents”, in TITS, 2022.

[5] T. Wang et al., “GSC: A graph and spatio-temporal continuity based framework for accident anticipation”, in TIV, 2023.

• Kaggle Competition (CVPR DriveX Workshop, 2025): As demonstrated in section B of our supplemental material, we compared our method with other participants' solutions in the Nexar Dashcam Crash Prediction Challenge [6], which is a Kaggle competition of accident anticipation on the Nexar dataset hosted by Nexar Inc., and the Top-3 solutions are announced on the CVPR DriveX Workshop, 2025. As shown in the table below, our method outperforms all the solutions from 282 participants and 237 teams by a large margin, further confirming that our method is state-of-the-art.

Table 2: Public and private scores of our method on the leaderboard of the Nexar Dashcam Crash Prediction Challenge [6].

[6] D. C. Moura et al., “Nexar dashcam collision prediction dataset and challenge”, 2025.

3. Theoretical Derivation and Analysis:

Thank you for your valuable suggestions. We will conduct an in-depth analysis of these two points.

• How far in advance can predictions yield maximum benefits:

In our paper, we set the prediction horizon to 2.0 seconds according to our experience. We acknowledge that this selection lacks theoretical analysis. To this end, we conduct comparison experiments of the prediction horizon (ranging from 1.0s to 3.0s). As shown in the following table, when the prediction horizon decreases to 1.0s, although the late anticipation performance improves a little, the ability of early anticipation degrades a lot. When the prediction horizon increases to 3.0s, the overall anticipation performance drops. Therefore, we come to the conclusion that setting the prediction horizon to 2.0 seconds yields maximum benefits.

• Dependency relationships between predictions at different time steps:

Our Transformer-based temporal decoder inherently models the dependencies between predictions at different time steps. The self-attention mechanism in the Transformer allows the model to capture complex, non-linear relationships across the entire predicted sequence. Each output probability is informed by the context of all other positions in the sequence, allowing for a holistic and interdependent prediction. We will revise the Method section of our paper to further elaborate on this dependency.

4. Fixed Probability Threshold for Warning Triggering:

You raised a critical point regarding the fixed probability threshold, and we agree that an adaptive mechanism is more aligned with real-world deployment.

• Determination of Fixed Threshold: Following the way of all previous works, the current fixed threshold was empirically determined based on achieving a balance between recall and false alarm rates on the validation set of the MM-AU dataset, aiming for an "acceptable" false alarm rate as per our refined evaluation protocol. This was a pragmatic initial choice for validating our method and a fair comparison with other methods.
• Adaptive Thresholding Mechanisms: We fully concur that a fixed threshold is a simplification for real-world scenarios. We propose to explore adaptive thresholding mechanisms in future work, potentially leveraging:
  • Context-aware Thresholds: Implementing different thresholds based on driving contexts (e.g., urban vs. highway, day vs. night, speed limits). This would require richer contextual information to be fed into the system or a context-aware classification module.
  • Uncertainty Estimation: Integrating uncertainty quantification into our probability predictions. Warnings could then be triggered not just by a high probability, but also by a high probability with low uncertainty, or by dynamically adjusting the threshold based on the predicted uncertainty for a given scenario.