When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network

Claims 3: V2E Transformation

Please refer to the response to Weakness in Reviewer rGL6 and Q1 in Reviewer WT2W.

Exp 3: Inference Speed

We evaluated inference speed on different platforms: RTX3090, RTX4090, and A100-80G. The RTX4090 achieved the fastest speed at 603 FPS, followed by the A100-80G at 579 FPS and the RTX3090 at 517 FPS. These modest differences demonstrate that our approach consistently delivers real-time performance across various hardware configurations.

Exp 4: Ablation Studies

Our ablation studies show that both the GRU and attention modules are critical. The GRU captures long-term dependencies to improve temporal feature aggregation and stability, raising the mTTA from 1.44 to 1.98 seconds. Meanwhile, the attention module focuses on key regions—especially when fusing RGB images with event stream data—ensuring the model prioritizes the most informative cues. Together, these modules enable our full model to achieve an AUC of 0.879, demonstrating their complementary benefits for anomaly detection.

Exp 5: Event Processing

We compared our asynchronous GNN with two lightweight methods: AsyNet, which uses sparse convolutions, and AEGNN, which employs asynchronous GNNs. Our experiments show that our approach achieves better detection performance with far lower computational overhead. Specifically, AsyNet uses 367 MFLOPs per event, AEGNN uses 10.98 MFLOPs, and our method only uses 8.732 MFLOPs per event. This efficiency and accuracy highlight the benefits of our approach for processing event-based data.

Ref: Related Work

In the revised manuscript, we will discuss recent advances [1,2] in asynchronous processing and multimodal fusion that address real-time constraints. For example, recent video-question answering work uses non-parametric probes like QUAG to decouple and evaluate both intra- and inter-modal interactions.

Weaknesses 1: Adverse Weather or Low-light Scenarios

I collected data on severe weather or low-light scenes on the ROL and DoTA datasets, and performed comparative experiments with the two best performing methods in the comparative experiments. Benefiting from the advantages of event cameras in extreme lighting scenarios, we surpass the performance of other methods by a large margin in this scenario.

Weaknesses 2: Failure Cases

Figure R3-1 shows that our system's performance relies heavily on the object detector's accuracy. In Frame25 and Frame30, blurry images and small objects led to a complete detection failure, which in turn caused the anomaly detection framework to fail. Although an object was detected in Frame35 when it was very close to the ego vehicle, the attention score was only 0.56 due to missing previous detections, preventing the GRU from building a robust temporal feature set. These cases highlight that our approach is limited by the reliability of the object detector, and improvements in detection accuracy are essential.

Suggestions 1: Visualizations

The Figure R3-2 shows that our target-level attention module assigns varying scores to bounding boxes. For instance, a vehicle cutting in starts with a low score (0.21 in Frame 20) that rises as it approaches (0.45 in Frame 25 and 0.71 in Frame 30). This indicates that our attention mechanism, combined with event stream features, effectively emphasizes fast-moving objects, leading to higher anomaly scores.

Q1: Asynchronous GNN

Our asynchronous GNN module requires only 8.732 MFLOPs per event—far lower than synchronous methods (74559 for Events+YOLOv3, 6984 for RED, and 27659 for ASTM-Net). This efficiency is achieved by processing events asynchronously, eliminating the heavy cost of frame-based operations.

Q2: Sensitivity Analyses

Our Figure R3-3 shows that the model remains robust across different settings. At the standard 0.5 IOU and 0.5 confidence thresholds, the model achieves an AUC of 0.879. As shown in Figure R3-4 and Figure R3-5, Increasing these thresholds sharply reduces AUC by filtering out more detection boxes, while lowering them causes a gradual decline as false positives are effectively down-weighted. Overall, the AUC ranges from 0.65 to 0.879, confirming the stability of our configuration.

We greatly appreciate your insightful feedback and practical recommendations. Detailed responses to your questions are listed as follows.

Methods And Evaluation Criteria 3: Two-stage Contribution

(1)Our two-stage model first performs object detection to generate bounding boxes, providing precise localization and object-level features for subsequent anomaly detection. In complex traffic scenarios—such as multi-vehicle interactions or crowded pedestrian areas—this initial detection stage enhances object recognition accuracy. (2)For example, when a vehicle changes lanes suddenly, the bounding box prior helps quickly lock onto the anomalous object, reducing false detections. (3)Moreover, in fast-moving or low-light scenarios where RGB images suffer from motion blur or poor illumination, asynchronous event streams capture fine luminance changes to compensate. This dual-modality support boosts detection robustness, as shown in our experiments. (4)In contrast, while a single-stage model can meet millisecond-level response requirements by sharing features and reducing redundant computation (achieving a higher FPS), it may sacrifice some accuracy. Thus, the two-stage model trades off increased latency for improved detection accuracy, making it well-suited for scenarios that demand high reliability, such as highway cruising.

Experimental 2: Computational Overhead

(1)Our method requires only 8.732 MFLOPs per event, uses 23.5 GB of video memory. Under typical conditions in the ROL dataset, with an average event rate of 560k EV/s, this overhead is manageable. In high-speed driving scenarios—where the event rate can rise to 1–10M EV/s—the worst-case computational load is approximately 87.32 TFLOPs. (2)Current autonomous driving chips such as Atlan, Thor, and Orin can fully deploy our algorithm on-board, while chips like Xaiver and Parker, although sufficient for normal driving, might face challenges under extreme conditions. The specifications of some representative chips are as follows:

References: Related Work

We acknowledge the importance of recent advances in multimodal and asynchronous processing outside the driving anomaly detection domain and we will cite these two articles in our paper. (1)For instance, [1] proposes a Recurrent Asynchronous Multimodal Network (RAM Net) for monocular depth estimation by combining event cameras and RGB cameras. This approach leverages ConvGRU units to support asynchronous inputs, preserving the high temporal resolution of event data while fusing rich spatial information, which enhances prediction accuracy in dynamic scenes. (2)Similarly, [2] introduces an Asynchronous Event Graph Neural Network (AEGNN) that efficiently processes sparse event data by performing local asynchronous updates within a spatiotemporal graph, significantly reducing computational overhead.

Q1: synthetic event data generation affect

(1)The current traffic anomaly detection dataset does not have a real Event mode. Our subsequent work is to use event cameras to collect this data.(2)V2E addresses this issue by considering various types of noise. First, it introduces temporal noise by utilizing Poisson noise to generate physically consistent noise events. Second, it simulates shot noise, which arises from photon statistics under low-light conditions in event cameras. Third, it accounts for threshold mismatch, where event trigger thresholds vary across different pixels in real event cameras, by modeling this phenomenon with a Gaussian distribution. For experimental verification of V2E, please refer to the response to Weakness in Reviewer rGL6.

Q2: Explainability of Abnormal Events

(1)RGB features correspond to appearance features, BBox features correspond to position and movement features, event stream features correspond to features of rapid and abnormal movement of objects, and object-level features are local detailed features. (2)Our model's object-level attention mechanism assigns scores to detected objects, highlighting those more relevant to anomaly detection. As shown in Figure R2-1, when a vehicle suddenly cuts in, its attention score increases as it approaches, indicating its growing importance as a potential anomaly. Visualizations of these attention scores across frames illustrate how the model dynamically focuses on critical objects, providing insights into the features it considers vital for identifying anomalies. This enhanced interpretability aids in understanding the model's decision-making process, thereby improving its credibility and deployment security.

Thank you for your inspiring review and actionable suggestions. Below, you will find our detailed responses to your questions.

Q1: Real-time Anomaly Detection Methods

We did omit some real-time anomaly detection papers, but these methods are not applicable to the traffic domain. For example, AED-MAE (CVPR 2024)[1] is a state-of-the-art anomaly detection method for surveillance videos that focuses on detection speed. However, it incorporates static background information into its approach and operates at the frame level. In autonomous driving, interference from dynamic backgrounds would significantly degrade its performance. Similarly, industrial anomaly detection methods such as EfficientAD(WACV 2024) [2] are also highly efficient but operate at the frame level, making them unsuitable for the complex environments of autonomous driving. MOVAD(ICASSP 2024)[3] is the first method to propose online real-time anomaly detection specifically for autonomous driving. We compare our approach with these three methods, and our method achieves the best balance between accuracy and speed, specifically designed to meet the unique requirements of autonomous driving. A key innovation of our work is its focus on achieving robust performance in dynamic environments. We tested these three models on ROL and DoTA and the results are shown in the following table.

Q2: Inadequate literature review

In fact, our comparison methods already include SOTA in traffic anomaly detection, such as MAMTCF (ArXiv 2023), AM-NET (IEEE TIV 2023), and TTHF (IEEE TCSVT 2024). In our revised manuscript, we will expand Table 1 to incorporate additional recent methods, including three state-of-the-art real-time anomaly detection approaches—EfficientAD, AED-MAE and MOVAD. We will clearly articulate the distinctions between existing video-frame-based real-time anomaly detection methods and our object-centric framework, which is specifically designed to address the challenges inherent in autonomous driving scenarios. Additionally, we acknowledge that our original claim that "most existing methods focus on accuracy but ignore speed" may have been too broad. In the revision, we will refine this statement to "most existing object-centric methods focus on accuracy but ignore speed" to more accurately reflect the current state of the art.

Q3: Methodological Novelty

Our method is the first to utilize both event streams and RGB data for traffic anomaly detection, making it a novel contribution. The key innovation of our approach lies in leveraging the unique characteristics of event streams as critical features for road traffic anomaly detection. Through graph-based modeling in asynchronous GNNs, we perform neighborhood aggregation only on active event nodes, combined with a lookup table acceleration mechanism in Spline convolution, enabling efficient feature extraction. Moreover, the continuity of event streams allows the model to perform anomaly detection between RGB frames (Figure 4), enabling early anomaly detection, which is another key innovation of this work. Finally, by incorporating global and object-level graph modeling along with a dynamic attention mechanism, the model leverages a GRU module to capture the temporal dependencies of these features, focusing on anomalous motion patterns of specific objects, thereby achieving accurate anomaly identification.

Q4: Figure 1

Figure 1 illustrates our anomaly detection process for autonomous driving. When a leading vehicle exhibits abnormal behavior, the model's anomaly score gradually increases until it surpasses the anomaly score threshold. Therefore, our primary focus is on whether the anomaly score can quickly reach the threshold when detecting an anomalous object.This process involves two key time components: $T_{\text{inference}}$, which represents the model's processing latency, and $\Delta T_{\text{detection}}$, the additional time required to recognize and confirm the anomaly—essentially, the time it takes for the anomaly score to reach the threshold. The sum of these components defines the overall response time, which reflects both the inference speed of our model and the latency in anomaly detection (i.e., the delay in quickly identifying an object at risk of being anomalous).

References

[1] Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors, CVPR 2024.

[2] EfficientAD: Accurate Visual Anomaly Detection at Millisecond-Level Latencies, WACV 2024.

[3] Memory-augmented Online Video Anomaly Detection, ICASSP 2024.

Thank you for your motivating review and concrete suggestions. Detailed responses to your questions are listed as follows.

Weakness: V2E Transformation vs. Real Event Stream

(1)The current traffic anomaly detection dataset does not have a real Event mode. Our subsequent work is to use event cameras to collect this data. (2)V2E effectively mimics key DVS sensor traits (Gaussian Threshold Distribution, Temporal Noise, Leak Events, Intensity-dependent Bandwidth). For example, on N-Caltech 101, using V2E data raised ResNet34 accuracy on real DVS from 81.69% to 83.36%, reaching 87.85% with fine-tuning (vs. 86.74% with real data alone). (3)DSEC[1] is an autonomous driving dataset and contains real Event modalities, but since it is a completely normal driving dataset, it cannot be used for our anomaly detection. In order to verify the validity of the data generated by V2E, We extended our evaluation using the DSEC dataset by generating a simulated dataset, DSEC+V2E. Training on DSEC for normal samples and testing on ROL and DoTA revealed only minor variations in anomaly detection accuracy, confirming the robustness of our model when incorporating V2E-generated event data. While there are the inherent differences between synthesized and real event data, our results clearly demonstrate that the V2E method effectively supplements event modality and maintains the robustness of the detection model. For more V2E issues, please refer to the Q1 in Reviewer WT2W.

Q1: Model Scalability and Real-Time Performance

(1)The ROL and DoTA datasets already include highly complex scenarios—extreme weather, nighttime conditions, intense lighting, and diverse camera perspectives. Despite these challenges, our model maintains excellent real-time performance and accuracy. (2)Our multimodal asynchronous hybrid network is designed modularly, allowing us to increase its depth (i.e., number of layers) to handle even more complex data. While deeper networks offer slight improvements in accuracy, they also introduce a small increase in processing delay. (3)Our experiments on the ROL dataset demonstrate the trade-off between accuracy and latency. In summary, our model’s modular architecture enables it to effectively adapt to increasingly complex environments with only a minor trade-off between improved accuracy and additional computational delay. (Layers contains a ResBlock and a look-up-table Spline convolution)

Q2: Transformer-based Architectures

Replacing the CNN backbone with a ViT or Swin Transformer enables global feature modeling and boosts detection accuracy by capturing long-range dependencies. However, Transformers add latency; for example, ViT’s self-attention has O(N²) complexity. Our experiments confirm that Transformer-based architectures improve accuracy when latency is managed.

Q3: Real-world Deployment

After calculation, our model can be fully deployed on most autonomous driving chips, and a small number of autonomous driving chips can be deployed through quantization. (1)Computing resource limitations: For instance, under normal conditions the system handles 560k events per second [2], and even at peak loads (up to 10M events/second), the overhead (~87.32 TFLOPs, 42W) is manageable on chips like Orin. (2)Although we need two camera devices, we can ignore the transmission delay issue. Hardware synchronization limits timing errors to ~78 microseconds, while event camera delays are around 6 ms. For more detailed deployment information, please refer to the response to Experimental 2 in Reviewer WT2W.

Q4: Different Vehicle Types or Camera Configurations

Our training data was collected from a diverse range of vehicles, including both sedans and trucks. Because our method is designed around an object-level approach that centers on the target detector rather than the camera or vehicle type, it naturally generalizes across different vehicle configurations. Actually, we used 20fps RGB camera data. However, the system can easily benefit from higher frame rate cameras in real-world deployments, potentially further enhancing detection performance without compromising the model's scalability.

References

[1] DSEC: a stereo event camera dataset for driving scenarios

[2] Prophesee Evaluation Kit - 2 HD