Ex-VAD: Explainable Fine-grained Video Anomaly Detection Based on Visual-Language Models

We sincerely thank the reviewers for their valuable comments. We will add these valuable comments to the revised manuscript.

R1: Evaluation of Inference Time. Thanks for your suggestion. We would like to point out that Table 7 in our main paper provides a comparison of relevant computational metrics, including Trainable Parameters, Inference Time, and Multiply-Add Operations (MACs). Our approach achieves similar inference speeds as previous methods while improving interpretability and fine-grained anomaly detection through VLM and LLM.

R2: Reasons to choose BLIP2. The use of VLMs and LLMs is an essential part of the solution to the lack of anomaly classification and anomaly explanations in existing studies. To reduce its required computational cost and resources, we tried various ways to generate captions during the experiment, such as BILP-2, LLAMA3.2, cogVLM, etc. We found that the optimal choice that combines both efficiency and performance is BLIP-2, which generates video captions with lightweight deployment and less time.

Regarding BLIP-2's intermediate features, we tried to integrate them into the model, e.g. by splicing them with video visual features. Experiments show that this method has an accuracy of 88.09% on the UCF-Crime dataset, while the anomaly explanation generated by LLM can achieve 88.29%. Therefore, we finally chose the explanation text generated by LLM which could describe the cause of the anomaly in addition to improving the accuracy by 0.2%.

R3: Explainable texts are not shown in the main text. Because of the 8-page limit, necessary experiments (such as accuracy of fine-grained and coarse-grained anomaly detection, ablation experiments, parameter selection, model complexity, and inference time) have taken up a major part of the paper, so we have placed the explainable text in the appendix section. An example showing the logic and results of anomaly text generation is given in Figure 3 in the main text. Additionally, we'll add more explainable text in camera-ready paper.

R4: More methods to compare. Thanks for your suggestion. We have compared our method with existing explainable video anomaly detection methods such as LAVAD[1], VERA[2], and VADor[3], and we will compare and discuss the suggested method Holmes-vad[4] in the camera-ready paper.

Different from Homles-vad which focuses on the coarse-grained anomaly detection task (determining whether a video frame is abnormal), our method can achieve coarse-grained anomaly detection and fine-grained anomaly classification (identify the abnormal category).

As Homles-vad requires more supervised information, additional introduction of WSVAD [5] to generate pseudo-labels as well as fine-tuning LLM, its coarse-grained anomaly detection results outperform our method. Three main reasons are as follows:

1. Holmes-vad incorporates more supervisory information, performs single-frame temporal annotation in constructing the dataset, and generates interpretable textual descriptions based on it, whereas our method stays consistent with the traditional video anomaly detection methods without any additional annotation.
2. Holmes-vad is based on the existing WSVAD [5] method which first performs prediction to get the pseudo-labels and then selects the frames with anomaly scores higher than the threshold, which focuses on the keyframes as compared to uniform sampling.
3. Holmes-vad needs to fine-tune the multimodal large language model and requires more computational resources for training. Experiments demonstrate that our method has only 9.97M training parameters and 15.37ms inference speed. Our method has a better balance between computational resources and effectiveness.

[1] LAVAD: Harnessing large language models for training-free video anomaly detection. CVPR2024

[2] Vera: Explainable video anomaly detection via verbalized learning of vision-language models. ArXiv2024.

[3]VADor: Video anomaly detection and explanation via large language models. ArXiv2024.

[4] Holmes-vad: Towards unbiased and explainable video anomaly detection via multi-modal llm. ArXiv2024.

[5] UR-DMU: Dual memory units with uncertainty regulation for weakly supervised video anomaly detection. AAAI 2023.

We sincerely thank the reviewers for their valuable comments. We will add these valuable comments to the revised manuscript.

R1: Robustness of the label. We use the SOTA large language model GPT to generate M phrases for label expansion and select the top-k among them as the final labels. To ensure the generated labels remain relatively stable, k is set to be much smaller than M. Experiments demonstrate that the best performance is achieved when k=4. This approach ensures a certain level of robustness in the quality of the generated labels.

R2: Analysis of Anomaly Explanation. Currently, evaluation methods for video-text tasks are more common in video understanding. Typical quantitative analyses compare the similarity between generated text and reference text using metrics such as BLEU, CIDEr, and METEOR. Additionally, qualitative analysis is conducted by comparing texts generated by different models, where higher relevance to the video indicates better quality. In the appendix, we provide a visualization analysis of our model, which includes interpretable text that, to some extent, reflects the model's understanding of the video. In the final manuscript, we will further supplement this aspect.

R3: More comprehensive comparison with other recent SOTA methods. You can see the reply to Reviewer VnC3's “R4: More methods to compare.” for a more detailed answer.

R4: Quantitative analysis of explainable texts. Quantitative analysis of explainable texts is our next step. Our next plan is to further standardize the process of anomalous explanatory text generation, and its qualitative analysis. Specifically, 1. optimize the text generation strategy: make the generated anomaly explanations more precise and readable by adjusting the prompt engineering or introducing more reasonable templates. 2. enhance the standardization of the qualitative analysis: introduce the qualitative assessment criteria of the text, such as BLEU, CIDEr, METEOR, etc.

We sincerely thank the reviewers for their valuable comments. We will add these valuable comments to the revised manuscript.

R1: Resource consumption. We appreciate your concern regarding the resource consumption of Ex-VAD due to the integration of VLMs and LLMs. We also recognize that due to the integration of multiple complex modules such as VLM and LLM, it may impose high resource requirements when deployed. Therefore, in the experimental session, we compared the training parameters and Multiply-Add Operations(MACs). Table 7 demonstrates that our method achieves a well-balanced trade-off between model complexity and size, optimizing both performance and resource usage effectively.

R2: Fine-tune the model. We appreciate your concern and suggestion about fine-tuning the model. At the beginning of our experiments, we thought that there were two ways to obtain anomaly explanations: the first way is to fine-tune the LLM to achieve the interpretability of anomaly videos, and the second way is to invoke the knowledge of the LLM by setting the cue words to obtain the desired anomaly explanations. Through the experiments, we found that the computational resources, model complexity, and reasoning speed required for fine-tuning LLM are much higher than that of the method of setting cue words when there is not much difference in accuracy. Therefore, ultimately, our approach is to efficiently utilize the general knowledge of LLM by setting cue words to achieve anomaly interpretation and improve detection accuracy.

R3: Multi-modal feature fusion during the training process. During the training process, we optimize and select the multimodal fusion strategy in the following two steps. Firstly, we refer to the fusion strategies commonly used by previous studies, such as contact, cross-modal attention, and addition, etc. Secondly, we try the above methods in the experimental process and select the optimal fusion method by comparing the accuracy.

R4: Future work. Thanks for your attention, our next plan is to further standardize the process of anomalous explanatory text generation, and its qualitative analysis. Specifically, 1. optimize the text generation strategy: make the generated anomaly explanations more precise and readable by adjusting the prompt engineering or introducing more reasonable templates. 2. enhance the standardization of the qualitative analysis: introduce the qualitative assessment criteria of the text, such as BLEU, CIDEr, METEOR, etc.

We sincerely thank the reviewers for their valuable comments. We will add these valuable comments to the revised manuscript.

R1：Novelty of the Proposed Pipeline. We apologize for failing to highlight our contributions and novelty. Different from existing coarse-grained VAD, our method is unique in using the generated anomaly text for both anomaly explanation and fine-grained anomaly detection. Compared with traditional fine-grained anomaly detection, our accuracy has a large improvement, e.g., in the UCF dataset, the average mAP(AVG) is 3.67% higher compared with SOTA. Additionally, compared with fine-tuning a large language model (LLM) or a multimodal large language model (MLLM) for anomaly detection, our approach requires significantly fewer parameters (9.19M) and achieves a faster inference speed (15.37ms). This demonstrates a good trade-off between complexity and effectiveness.

R2：Reasons for not using ShanghaiTech and UCSD-Ped2. The key idea of our method aims at the fine-grained detection and description of the causes of anomalies, and its advantage is to generate anomaly explanation text through VLMs and LLMs, which both describe the causes of video anomalies and greatly improve the accuracy of the fine-grained anomaly classification. Therefore, we select datasets that contain fine-grained anomaly detection in existing studies, such as UCF-Crime and XD-Violence, whereas the Shanghai and UCDF datasets have not been used by researchers for fine-grained anomaly detection, and thus cannot be compared with the superiority of our method.

Considering your suggestion, we will add related experiments for the shanghaiTech and UCSD-Ped2 datasets. Due to time constraints, we only had time to add the results of the coarse-grained anomaly detection experiment on shanghaiTech, and will add the results on UCSD-Ped2 later. As shown in the below table, we performed coarse-grained anomaly detection on the ShanghaiTech dataset and compared it with other methods:

Although our method is not SOTA, it is only 0.12% less than the best result, 0.13% higher than TEVAD, and 0.42% higher than STPrompt, which also uses the clip to extract features. This experiment shows that our method is also applicable to the Shanghai dataset.

R3: Citation and discussion of TEVAD. TEVAD makes full use of VLM to generate video captions as textual modalities and achieves improved accuracy in anomaly detection by fusing it with visual modalities. We will cite and discuss it in the final manuscript.

Both TEVAD and our method utilize VLMs to generate captions, which extends the traditional single visual modality to visual and textual multimodal branching, improving anomaly detection. Our advantages over TEVAD are shown below. First, our proposed method not only determines whether a video frame is abnormal but also further identifies the abnormal category (Abuse, Arrset, etc.). Second, instead of just generating captions, the video anomalies are further leveraged to describe the video anomalies through LLMs.

R4: others. Thank you for pointing out the spelling error, we will change Table 6: 'Lable Augmnet' to 'Label Augment' in the camera-ready paper.