Learning Event Completeness for Weakly Supervised Video Anomaly Detection

Reviewer zcbm

Q1: The formulation of the local transformer and global GCN needs clarification. What are the main contributions of these modules? how does it affect the proposed approach?.

We appreciate the reviewer's insightful suggestions. The CLIP image encoder is primarily used to extract image features. But it exhibits limited proficiency in modeling temporal dependencies inherent in videos. To address this limitation, we integrate a local transformer layer alongside GCN modules to augment video features. We conduct ablation studies on GCN and Transformer modules and report the fine-grained AVG performance on XD-Violence:

In detail, we incorporate one local Transformer layer and 4 GCN layers. We present ablation studies that explore the rationale behind our choice and report the fine-grained AVG performance on XD-Violence:

Also, we report fine-grained AVG results on UCF-Crime:

Q2: In the loss function formulation, only two loss functions are evaluated and presented in the manuscript. What about the other loss functions?

Thanks for your comments. We do some ablation studies to evaluate the utilities of $L_{gmm}$ and $L_{reg}$ in Table 6. For the $L_{agnostic}$ and $L_{aware}$, since our model is designed to predict coarse-grained and fine-grained anomalies simultaneously, $L_{agnostic}$ and $L_{aware}$ are the basic terms used to train the model. Specifically, $L_{agnostic}$ is utilized to supervise coarse-grained anomaly detection, and $L_{aware}$ is utilized to supervise fine-grained anomaly detection. If either of them is removed, the model cannot detect anomaly events of the corresponding granularity (coarse-grained and fine-grained).

Q3: In Table 5, what are the differences between the last rows and the second-to-last row in terms of experimental settings? If there are different settings, how did the authors evaluate them?

Thanks for your comments. The last row of Table 5 indicates that the proposed algorithm leverages both visual information and textual descriptions of anomaly categories for both coarse-grained and fine-grained detections, while the second-to-last row only uses the visual information of the video. There is no difference in the experimental settings of other parameters.

Q4: As shown in Figure 2, there are three branches that are components of the proposed approach. However, in the ablation section, only VOB and CMB are evaluated. What about the other branch?

Thanks for your suggestions. In the ablation section, we evaluated the VOB and CMB, and the other branch is the proposed Gaussian Mixture Prior-based Local Consistency Learning mechanism, which is only used to guide the model's training. We explore ablation studies about this module by controlling the loss term $\mathcal{L}_{gmm}$ in Table 6. For clarity, we add several experiments and put them in a table for comparisons:

Q5: The authors need to discuss some of the related work in the introduction section to strengthen the motivation for the proposed approach.

We appreciate the reviewer's insightful suggestions. We will strengthen the discussion about our motivation in the camera-ready version. Here, we also re-emphasize it. We observe the existing methods predict incomplete and fragmented event segments, and an example is also provided in Figure 1. The core reason for this phenomenon is the lack of frame-wise annotations in the weakly supervised learning framework, resulting in sparse semantics. For this reason, we propose a novel Gaussian mixture prior-based strategy and memory bank-based prototype mechanism to mitigate this issue.

Q6:The authors need to carefully review the figures and table captions to ensure clarity for the reader".

Thanks for your suggestions. We will modify these typos and inaccurate expressions.

Reviewer MBS1

Q1: The dual structure has already been proposed in VadCLIP, so it would be preferable not to present it as a primary contribution.

Thanks for your suggestions. We will update it on the camera-ready version.

Q2: The ablation experiments use the setting where only one module is ablated independently, which is not ideal for verifying the relationships between each contribution. It is recommended to provide ablation experiments that incrementally introduce each module.

We appreciate the reviewer's insightful suggestion. Here, we provide an extra ablation study on XD-Violence to verify the relationships between each contribution. We have observed that the proposed modules can bring positive gains.

Q3: Multiple dense anomalous events may exist in an anomalous video, such as some testing videos in XD-Violence, and it is important to evaluate whether the proposed method might incorrectly classify these frequent anomalies as a single anomalous event.

We appreciate the reviewer's insightful suggestion. For fine-grained anomaly detection, the model is asked to predict anomalous events about each category, as reported in Table 3 and Table 4. Here, we divide the original test set (800 videos) into two parts. One part contains videos with only one type of abnormal events, and the other part contains videos with multiple types of abnormal events. The former contains 753 videos, while the latter contains 47 videos and is more challenging. We report fine-grained prediction results in these 47 videos that contain multiple anomaly categories:

We observe that our proposed method achieves significant improvement, especially when the the evaluation criteria are more stringent, namely larger tIoU values.

Q4: The visualizations in Figure 3 should provide more details, such as the source of the visualized features, particularly clarifying whether the visualized normal features include those from normal segments within anomalous videos. Meanwhile, this visualization demonstrates the discriminative visual representations for different anomaly categories. However, in the event completeness visualization presented in Figure 5, which is a binary classification result, the analysis lacks the explanation of how the visual representations for different anomaly categories contribute to enhancing event completeness.

We sincerely appreciate the reviewer's insightful comments and constructive suggestions regarding the visualizations in our paper. Below, we address the specific concerns raised about Figure 3 and Figure 5.

For Figure 3, we acknowledge the reviewer's request for clarification on the source of the visualized features. The visualized features are the enhanced feature $X_{l}$ learned by our model, and the source of them are features extracted by the CLIP image encoder. Besides, the visualized normal features include those from normal segments within anomalous videos. As shown in Figure 3, these features form clusters well, further proving that our learned features are highly discriminative. We will revise the caption of Figure 3 for these suggestions in the camera-ready version.

Although Figure 5 highlights binary classification results, the model utilizes the distinctive representations learned for different anomaly categories (depicted in Figure 3) to inform its decisions. This capacity to differentiate among anomaly types indirectly helps delineate the boundaries and attributes of anomalous events.

Furthermore, In the anonymous link https://anonymous.4open.science/r/Visualization_ICML_Rebuttal-B61B/README.md, we also provide some visualization examples to analyze the influence of different anomaly categories on enhancing event completeness.

Q5: Some other suggestions about expressions and the section structure.

Thanks for your valuable suggestion. We will improve the structure of Section 3 and some expressions in this paper in the revised version.

Q1: The fine-grained detection assumes closed-set categories. However, the reliance on category labels in VAD may limit the generalizability, since most abnormal categories are unknown/unpredictable.

Thanks for your suggestion. We agree that an over-reliance on category labels could potentially limit the generalizability. However, our proposed two mechanisms can mitigate the risks, including the usage of the CLIP text encoder and the prototype-based memory bank mechanism (PMB). First, CLIP itself can recognize open-set categories. Second, PMB can be understood to some extent as identifying general patterns and characteristics of anomalies, instead of focusing on learning specific categories.

To verify the generalization, we report the AUC on an extra dataset UBnormal, where only 7 abnormal categories are visible during training and 12 abnormal categories are used for testing. The results show our model can efficiently handle open-set anomaly categories, especially when using PMB.

[1] Not only look, but also listen: Learning multimodal violence detection under weak supervision. ECCV

Q2: The Gaussian mixture mechanism primarily enforces local consistency rather than explicitly addressing completeness. Visualizations of Gaussian-rendered scores would strengthen this claim.

Thanks for your suggestions. We define the average tIoU between predictions and GT for all classes' instances as a metric of the Event Completeness (EC) and report fine-grained results on XD-Violence:

We offer visual comparisons that illustrate the differences in outcomes when incorporating Gaussian-based scores versus not using them in https://anonymous.4open.science/r/icml_vis-CDD6/README.md

Besides, we also provide visualizations for various categories in https://anonymous.4open.science/r/Visualization_ICML_Rebuttal-B61B/README.md We find that our model can get desirable results.

Q3: The proposed Gaussian mixture and memory-based prototype techniques.

We appreciate the reviewer's insightful observations. First, both [1] and [2] are used in VAD instead of WS-VAD. Besides, they only model a unimodal Gaussian model with complex kernel function designs, but our method models prediction scores of multiple anomaly categories as Gaussian Mixture Model to ensure local consistency of predictions. With GMM, our model can learn the correlations between multiple anomaly categories, but the unimodal Gaussian model cannot.

For the memory-based prototype technique, [3] uses point annotations of action clips to create a memory bank storing reliable visual prototypes with threshold strategies. However, our weakly supervised scenario cannot access point annotations, so our memory-based is designed to build flexible text representations of anomaly categories. This can be also understood as general patterns of learning abnormal behaviors, as we explained in Q1. Besides, [4] learned prototype-guided discriminative embeddings to separate normal and abnormal data. It uses a different paradigm and purpose, which is essentially different from our method.

Q4: The absence of coarse-grained ablation studies weakens the evaluation.

Thanks for your suggestions. Due to the page limit, we showed more challenging fine-grained results in the original paper. Here, we provide coarse-grained results using I3D features on XD-Violence

[1] Ablation studies of the model structure:

[2] Ablation studies of the loss function:

[3] Ablation studies of text enhancement:

We observe that coarse-grained results show consistent conclusions with fine-grained versions, which further reveals the utilities of the proposed components.

Q5: Multi-label evaluation for XD-Violence dataset.

We use the same evaluation paradigm and codes as prior works like VadCLIP, ReCLIP, etc. Among 800 test videos, 47 samples have multiple labels. As labels for individual anomaly events aren't provided, the current evaluation paradigm only computes mAP without considering classes, which may introduce bias. However, since the proportion of such multi-label cases is low and all current methods follow this paradigm, the comparison remains fair. We also evaluated results only on 47 videos and found that its results were far lower than those on 800 videos, which reveals its results did not play a leading role.

Reviewer QCZn

Q1: The impact of the local transformer layer and the GCN module in enhancing video features.

Thanks for your advice. The CLIP image encoder is primarily used to extract image features. However, it exhibits limited proficiency in modeling temporal dependencies inherent in videos. To address this limitation, we integrate a local transformer layer alongside GCN modules to augment video features. In detail, we incorporate one local Transformer layer and 4 GCN layers. In the following table, we present ablation studies that explore the rationale behind our choice and report the fine-grained AVG performance on XD-Violence.

Also, we report fine-grained AVG results on UCF-Crime:

Q2: The rationale behind the choice of 𝐾 in top-𝐾 scores

In this paper, we set $K=max(\lfloor T/16\rfloor$), as explained in Implementation Details. This decision stems from our utilization of multi-instance learning (MIL) to get video-level outcomes. Previous works that used MIL used the same setup [1,2], which can adapt to the length of videos. The denominator uses 16 because these two datasets sample 16 consecutive frames into one segment during feature preprocessing.

[1] Two-Stream Networks for Weakly-Supervised Temporal Action Localization with Semantic-Aware Mechanisms, CVPR 2023 [2] Vadclip: Adapting vision-language models for weakly supervised video anomaly detection, AAAI 2024

Q3: The inference time of the proposed method

Thanks for your advice. I provide the average inference time (in seconds) of each video for fine-grained predictions on XD-Violence and UCF-Crime, and compare results with representative methods with a single Nvidia V100 GPU:

We observe that our method outperforms the SOTA ReFLIP in terms of speed. When compared to VadCLIP, it incurs only a marginal increase in processing time, yet achieves substantial improvements in performance.

Q4: The definitions of VOB and the CMB.

Thanks for your comments. VOB denotes both coarse-grained and fine-grained predictions derived exclusively from visual information, without incorporating category text descriptions. Instead, CMB combines visual inputs and text descriptions of anomaly classes.

Q5: Eq. 8 is chosen to regularize the predicted scores rather than ground truth.

Thanks for your comments. Eq.8 acts as a regularizer on the prediction scores and should not be used for GT. We hope to constrain the consistency of coarse-grained and fine-grained predictions by using Equation 8

Q6: The C3D result for LEC-VAD is missing in Table 1.

Thanks for your advice. We apologize for the omission of this result in the original paper. To rectify this, we will include the C3D results for our method in Table 1. This result is also presented here:

Q7: Discussions about publications including LLM- or VLM-based approaches.

Thanks for your advice. Some LLM- or VLM-based methods are used to generate abundant and diverse anomaly class descriptions. Notably, these works introduce additional data and knowledge provided by LLM, and the results based on them may be unfair comparisons. This introduction of external expert knowledge is not in the scope of this article. However, we will add relevant works and discussions in the related work.

Q8: The concept of Event Completeness and how our method addresses this aspect.

Event completeness can be conceptualized as the proportion of predicted intervals relative to the total actual event intervals. It can be reflected when using larger tIoU values. As shown in Table 3 and Table 4, compared with ReFlip, our method achieves more relative improvements for larger tIoU values.

Q9: A detailed comparison between the proposed method and ReFLIP-VAD.

The core idea of ReFLIP-VAD includes: 1) modeling the global and local temporal dependencies; and 2) designing learnable prompt templates to provide interpretable and informative class clues. However, our method focuses on the event completeness. While our focus is also on generating informative text descriptions, we introduce an innovative memory bank-based mechanism, eschewing the use of prompt templates generated by an additional pre-trained encoder.

Q10: Consider enlarging the text in Figure 3 for improved readability.

Thanks for your advice. We will improve them in the camera-ready version.