SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations

Dear Reviewer PM58,

Thank you so much for your valuable suggestions to help improve our paper! We deeply appreciate your recognition of the novelty and effectiveness of our work! Below, we have provided a detailed, point-by-point response to your questions and hope these can help address your concerns.

W1: Dataset Construction: (1) The release and separation of the dataset is a concern. The author can only provide links to publicly available sources and annotations. But it is common that the link may fail.I understand that this is something unavoidable, but it will undoubtedly reduce the frequency of use and impact of this dataset. (2) 2M videos for human verification is a huge effort, the authors don't provide any details of the procedure.

We sincerely appreciate the reviewer’s concerns regarding the public impact of our dataset Firstly, We acknowledge that links to publicly available sources may occasionally fail, potentially reducing the dataset's usability over time. However, this approach is necessary given the large scale and safety-related nature of this dataset. It aligns with the practices of several widely adopted datasets in the CV and NLP communities such as LAION [1] and Kinetics [2], which also provide links to publicly available sources instead of directly hosting the data. This approach is particularly common in video datasets, where the storage requirements are significantly larger. For example, Kinetics provides only URLs and invites practitioners to download videos using provided scripts, a practice we have followed for similar reasons. Furthermore, while our proposed guardrail model SafeWatch has been trained on a dataset containing sexually explicit, harmful, and violent content, we will not release SafeWatch-Bench in its entirety. Instead, we will release a carefully curated subset that meets the following criteria: (1) Extremely unsafe content, such as instances of child abuse, has been removed; (2) All human identities have been blurred to protect privacy; (3) Only content permitted under copyright laws is included, ensuring the data cannot be misused. For the releasable subsets, we are considering maintaining a local copy of certain data and providing direct access to the videos in a manner that complies with legal regulations. This will ensure the reproducibility of our dataset while preserving the integrity and impact of our work. We will ensure to handle this carefully and detail it in our dataset impact statement following your suggestions.

Regarding the second question, we totally agree with the reviewer that 2M videos for human verification are a huge effort, and in practice we do not ask human verifiers to review all 2M videos, instead, we only ask them to review one video in each batch of the 200K videos that we used for training the model, as well as all the 1420 videos in the test set that we used for evaluation, keeping the manual verification effort manageable. For the remaining videos in the 2M dataset, annotations are generated exclusively using the multi-agent pipeline without human verification. While this may introduce a certain error rate in the remaining dataset, we have demonstrated the effectiveness of our closed-loop pipeline in safety-related video annotation below that can effectively ensure the overall dataset's quality.

We provide detailed statistics of our dataset in Table-r. 1 below and more details regarding the data curation process in Table-r. 2. Specifically, we first group the 200K videos based on similar sources and types (e.g. same user ID or benchmark subcategory) in a batch (we use a batchsize of 64); (2) Then we run the multi-agent discussion pipelines event-by-event to annotate each video in the batch (all prompts provided in Appendix.B8); (3) Then we ask human verifiers to sample one video from each batch to review their explanation quality and decide whether to reject the batch and re-annotate. Specifically, grouping similar videos in a batch ensures they have similar annotation quality or shared issues, improving efficiency and reducing manual costs. If a batch has been rejected twice, then we discard this batch to exclude it from the dataset. As shown in Table-r. 2, human verifiers validate 3247 batches in total, with a low first-time rejection rate of 13.79%, demonstrating the effectiveness of our pipeline. Among the re-annotated batches, 23.7% (6784 videos) were rejected again and discarded to ensure the overall quality of the dataset.

[1] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., ... & Jitsev, J. (2022). Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35, 25278-25294.

[2] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., ... & Zisserman, A. (2017). The kinetics human action video dataset. arXiv preprint arXiv:1705.06950.

Table-r. 2 Additional statistics of the SafeWatch-Bench Dataset. We provide details for both training and testing configurations, including total videos, average video length (seconds), explanation length (word count), number of events, and the ratio of multi-labeled videos.

Table-r. 3 Performance of SafeWatch during each training stage

W3: Reproducibility Concerns: Reproducibility in data collection and model training is questionable. For instance, Section 4.2 on "multi-agent consensus video annotation" provides a basic idea of the processes but lacks sufficient detail for replication (e.g., missing the prompts used, configurations such as the number of frames used for each model, etc.). Additional issues are noted in the "Questions" section.

Thank you for this valuable suggestion! We have provided more details regarding the multi-agent discussion pipelines to better demonstrate the quality of our annotation results. Specifically, we uncover the details of the multi-agent discussion pipeline from five perspectives:

• Annotation Procedure: (1) we first group the collected videos with similar sources and types (e.g. same user ID or benchmark subcategory) in a batch (we use a batchsize of 64); (2) Then we run the multi-agent discussion pipelines event-by-event to annotate each video in the batch (all prompts for the three phases, i.e. proposal, discussion, and judge are provided in Appendix.B8). We use six surrogate MLLM-based agents for annotation and provide the configuration in Table-r. 5 below. (3) Then we ask human verifiers to sample a subset from each batch to review their explanation quality and decide whether to reject the batch and re-annotate. Specifically, grouping similar videos in a batch ensures they have similar annotation quality or shared issues, improving efficiency and reducing manual costs. If a batch has been rejected twice, then we discard this batch to exclude it from the dataset.
• Effectiveness: As shown in Table-r. 4, human verifiers validate 3247 batches in total, with a low first-time rejection rate of 13.79%, demonstrating the effectiveness of our pipeline. Among the re-annotated batches, 23.7% (6784 videos) were rejected again and discarded to ensure the overall quality of the dataset.
• Efficiency: The multi-agent pipeline iterates in a close-loop manner through three phases, i.e., proposal, discussion, and judge to gradually reach a high-quality annotation. The results in Table-r. 4 denotes that our pipeline can efficiently produce a high-quality annotation for most unsafe videos in 1-2 iterations, while the rejected videos incur more iterations due to the videos are more ambiguous and harder for the agents to achieve consensus.
• Human Perspective Alignment: We mainly guarantee the quality of the explanations through a close-loop multi-agent discussion and judge feedback, and further ask humans to select those explanations that align with their values during verification. To quantitatively verify the alignment with the human perspective, we design an experiment where we split 20 batches not used during training and prepare a pair of responses for each video where one is the final annotation resulted from our pipeline and the other one is directly using GPT-4o to provide the annotation. Then we adopt the following two metrics:
  • Implicit reward from the preference-aligned SafeWatch model, ranked by the log-likelihood ratio: $\log \frac{\pi(y_1|x)}{\pi_{\text{ref}}(y_1|x)} > \log \frac{\pi(y_2|x)}{\pi_{\text{ref}}(y_2|x)}$
  • Rankings provided by human reviewers

The results are shown in Table-r. 6, which indicates that both the DPO model and human verifiers preferred annotations from our pipeline over GPT-4o in approximately 90% of cases, validating strong alignment with human preferences.

Dear Reviewer YtKs,

Thank you so much for your valuable suggestions to help improve our paper! We deeply appreciate your recognition of the novelty of our work and its technical contribution to addressing the risks brought by the rapid proliferation of MLLMs. Below, we have provided a detailed, point-by-point response to your questions and hope these could help address your concerns.

W1: Missing Evaluation: The evaluation of the Safety-aware Event Sampling step is absent. This step should be crucial for model performance, as the authors used TransnetV2 to segment videos into safety-aware events, sampling a single frame per event for further MLLM processing.

Thank you for this valuable suggestion! We totally agree with the reviewer that the safety-aware event sampling model is a crucial component for the guardrail performance, and we have provided an additional evaluation of it below in Table-r. 1. Specifically, to reduce the heavy annotation workload, we first observe the connection between safety event segmentation and shot boundary detection, where we find that while being similar, our task differs in that multiple consecutive shots can belong to the same event. Noting this difference, we first adopt a SOTA shot boundary detection model AutoShot [1] to perform an initial segmentation on 742 videos sampled from the SafeWatch-Bench training set. These videos were carefully selected to ensure a comprehensive representation of all the unsafe video categories. Next, we ask human verifiers to review the segmented results and make corrections when necessary (primarily merging segments). This approach allowed us to produce sufficient high-quality frame annotations tailored for the safety event sampling task. Then we further split 74 videos as a test set and followed AutoShot to train our model based on TransnetV2. The evaluation results are presented in Table-r.1. Specifically, our model outperforms other models on the safety-aware event sampling task in terms of F1 score. Notably, our model achieves much higher precision compared to other general shot boundary detection models, reflecting its suitability for this specific safety guardrail task.

Table-r. 1 Evaluation of Safety-aware Event Sampling Model

We hope this additional result and explanation have addressed your concerns, and we have updated the revision following your suggestions.

[1] Zhu, W., Huang, Y., Xie, X., Liu, W., Deng, J., Zhang, D., ... & Liu, J. (2023). Autoshot: A short video dataset and state-of-the-art shot boundary detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2238-2247).

Q1: Could the authors design corresponding experiments and proofs to demonstrate that the mechanism producing the algorithm's effects aligns with their claims that the PEPE algorithm can "allow each policy to be encoded independently and in parallel" and that "equivalent positional embedding ensures that different policies are treated without bias"?

Thank you for this thoughtful question! In response, we have designed two additional experiments to separately verify that PEPE allows each policy to be encoded independently and in parallel and equivalent positional embedding ensures that different policies are treated without bias. We provide the additional evaluation results in Table-r. 1 above and also in Fig.6, 7 in the updated appendix. We also provide a theoretical analysis from a causal learning perspective to prove the effect of PEPE in our response to W1. For more details, please refer to our response to W1.

Q2: Could the authors provide some quantitative analysis of the annotation quality? Can this pipeline approach human-level annotation quality? Compared to annotation by a single LLM/VLM, what advantages does incorporating Multi-agent Discussion bring?

We really appreciate the reviewer's concern about the annotation quality of our pipeline, and we further uncover the effectiveness of the multi-agent discussion pipeline from five perspectives in our response to W2. Specifically, we provide detailed explanations regarding the annotation procedure, effectiveness, efficiency, human perspective alignment, and several case study to explain how this approaches human-level annotation quality. Essentially, we ask human verifiers to validate the quality of a subset of each batch of the annotation and the results in Table-r. 2 indicate over 86% annotated batches were accepted by human verifiers. We further design a specific experiment where we use an automatic model and ask human verifiers to compare the quality of the annotations resulting from our pipeline and GPT-4o, and we find that the automatic metric and human prefer our annotation 90.23% and 85.39% of the time.

Specifically, we conclude the reason that multi-agent discussion can achieve human-level annotation quality to be three-fold:

• Comprehensiveness: Multi-agent discussion integrates the capabilities of different LLMs/VLMs, where they can compensate with each other to provide a more comprehensive and in-depth understanding of the video. Furthermore, our discussion mechanism encourages agents to more logically and thoroughly reason about the contents in the video in order to convince other agents.
• Consistency: By encouraging agents to reach consistency through discussion, we can effectively search for a commonly agreed intersection of each agent's opinion, ensuring the accuracy of the final annotation.
• Close-loop feedback: By integrating a more powerful judge model to review the dicussion and provide feedback, we can gradually correct errors and improve the accuracy and quality of the annotation through iterative discussions. In contrast, the quality of single LLM/VLM annotations cannot be guaranteed as they lack this iterative refinement and error correction mechanism.

Ethics Concerns: The example in the upper left corner of Figure 3 may have 2 issues. (1) This type of data likely comes from real copyrighted videos, which may involve copyright infringement; (2) The authors did not blur or mask faces in their examples, which could raise privacy concerns.

We thank the reviewer for raising these concerns. In response, (1) We will release a subset of the dataset that will not contain copyrighted videos. As for concerns related to the release of the models, since this model is released for exclusive research purposes and implies a transformative usage of the video data (our model uses the video to produce improved textual descriptions, and cannot reproduce the video itself since it is not a video generative model) this usage of copyrighted data falls under the fair use doctrine. For further compliance, we will impose licensing that will prevent misuse of the released models. (2) We will ensure that all human identities, including faces, are blurred or masked in both the examples and the released dataset to mitigate any potential privacy issues.

We hope the additional details about our dataset and training pipeline, along with the expanded explanations, have addressed your concerns. Please don’t hesitate to let us know if there’s anything else you’d like to discuss or if further clarification is needed.

If all your concerns have been resolved, we would greatly appreciate it if you could consider raising the evaluation score to help us share this work with the broader community.

Thank you again for your time and thoughtful feedback!

Best regards,

Submission #9949 Authors

Dear Reviewer yJiC,

Thank you very much for these valuable suggestions to help improve our paper! We are really grateful for your appreciation of the novelty of our work and its contribution to enhancing the trustworthiness of video generation and video security! Below we have provided a point-by-point response to your questions and hope they could help address your concerns.

W1: The working mechanism of the PEPE algorithm lacks detailed theoretical explanation or experimental validation. The authors conduct ablation experiments to prove the effectiveness of the PEPE algorithm, but they don’t provide sufficient proof of its underlying principles. In lines 293-297, the authors claim that the PEPE algorithm can provide independent representations for each policy, which can alleviate the position bias problem in MLLM mentioned in lines 266-269. However, regarding this claim, there are neither experimental designs nor mathematical proofs to support it. I have doubts about whether the mechanism behind the algorithm truly aligns with the authors' claims.

We totally agree with the reviewer that a more in-depth analysis is required to validate the underlying effect of PEPE. Therefore, we have designed two additional experiments to separately prove that PEPE allows each policy to be encoded independently and in parallel and equivalent positional embedding ensures that different policies are treated without bias. Specifically, we provide the additional evaluation results in Table-r. 1 below and in Fig.6, 7 in the updated appendix. We detail our experiment design and findings point-by-point.

• Independent, parallel policy encoding: We design the first experiment by permuting each policy across different positions in the input and analyzing their attention score. Ideally, we would expect the attention score to be invariant to the policy position with independent, parallel encoding and have a constant attention score for each policy. Specifically, we randomly select a video flagged by both Sexual and Violence and depict the attention score curves in Fig. 6 in the appendix. The results indicate that the policy attention scores of SafeWatch indeed preserve constant, verifying that PEPE has eliminated the policy interdependency by decomposing the policy guidelines into several independent blocks and applying them with equivalent position embedding. We note that while the curves are not perfectly constant due to a pair of special tokens in between each policy (which is position-sensitive), which might incur some unavoidable but small interdependent patterns that can be omitted as a structural noise. In contrast, InternVL2-8B showed a very strong positional bias where the policies in earlier position tend to have higher attention weights in general. The curves of the model without PEPE also indicate that policies permuted among different positions may result in completely different attention scores, further indicating severe interdependencies between policies in the absence of PEPE. By independently encoding policies this way, PEPE effectively eliminates the spurious interdependency between policies and enhances the robustness of the guardrail result.
• Equivalent positional embedding eliminates bias: We design another experiment by investigating the correlation of the policy attention score with both the policy position (represented by linear line vector) and the policy category (represented by one-hot vector) over the SafeWatch-Bench dataset. We evaluate the correlation with both Pearson Correlation Coefficient (PCC) and Spearman’s Rank Correlation Coefficient (SRCC), and we provide the additional evaluation results in Table-r. 1 below and Fig. 7 in the appendix. Specifically, the policy attention score encoded with PEPE showed a very low correlation to the policy position ($\leq1\%$), and a reasonably strong correlation to the correct policy category. This means that given a video input which violates a specific policy, the attention corresponding to that policy is higher, which aligns with our expectations. This demonstrates that PEPE effectively mitigates positional bias while improving the model's interpretability. In contrast, models without PEPE showed a strong correlation between policy attention scores and policy position, while being largely irrelevant to the correct policy category. This highlights the presence of significant positional bias in those models. Furthermore, our findings indicate that increasing the model scale does not mitigate this bias effectively. In summary, PEPE has proven to be an effective approach to addressing positional bias, ensuring higher interpretability by aligning attention with the correct policy category.

Dear Reviewer QsGZ,

Thank you very much for your valuable suggestions to help improve our paper! We deeply appreciate your recognition of the novelty of our work and its critical contribution to handling the risks brought by the rise of MLLMs. Below, we have provided a detailed, point-by-point response to your questions, and hope these could help address your concerns.

W1: Details about the training and testing splits within the benchmark are insufficient, leaving questions about data partitioning.

We totally agree with the reviewer that the details on training/testing splits are critical for understanding the mechanism and effectiveness of our work. In response, we have provided additional details on the dataset partitioning and statistics in Table-r. 1 below, as well as Fig. 10-20 in the updated appendix. Specifically,

• Dataset partition: For more effective training and evaluation, we use the 200K videos verified by humans via batch sampling and select 1420 videos to consist the testing set for benchmarking (830 real-world videos, 590 generative videos), and use the rest of 199604 videos for training. Specifically, we aim to ensure the diversity of the videos and a balanced coverage of all categories in the test set (more detailed distribution is shown in Fig. 10, data sample case study in Fig. 16, 17, 20 in the appendix).

• Data usage in each training stage: We detail the data usage for each of the three fine-tuning stages, and we demonstrate the corresponding model's performance in each stage in Table-r. 2:

  • Multi-task Guardrail Training: In this stage, we randomly sample 80K guardrail-related videos we collected and 30K normal videos in ShareGPT4Video annotated by GPT-4v, and then we augment the original annotations into multiple tasks including captioning, VQA, and video guardrails, resulting in 220K training samples. This stage aims to train the model to develop general guardrail capabilities while preserving a broad understanding of general video content, effectively mitigating catastrophic forgetting and overfitting to guardrail-specific videos.
  • Adaptive-Pruning Training: In this stage, we solely fine-tune the model on all the 199K guardrail-related videos using four types of guardrail task prompts specified in Appendix.B8. This stage aims to train the model to extract essential information from a subset of more informative video tokens via PAP and downstream the model for specialized guardrail tasks.
  • Preference Post-tuning: In this stage, we aim to further improve the quality of explanations. Specifically, we curate the rejected explanations from two sources (1) offline collection: the non-specific or overly long explanations that we discarded during the multi-agent propose-discuss pipeline; (2) online sampling: we run the model from the previous stage to infer through 5K diverse videos in the training set and collect those samples with wrong answer. And we use the corresponding ground-truth explanations as the chosen pair. This process results in 60K problem-centric preference pairs and we fine-tuned the model using DPO.

• Additional Dataset Statistics: We provide more statistics to better understand our dataset and training procedure. Specifically,

  • The average video length in the training and testing set is 57.49 and 61.12 secs, respectively (more detailed distribution in Fig. 11 in the appendix), with the longest video up to 90 minutes, ensuring a comprehensive coverage of unsafe videos with different scales (e.g. short videos, movies).
  • The average explanation length in the training and testing set is 80.2 and 73.16 words, ensuring detailed and in-depth reasoning of the guardrail result (more detailed distribution in Fig. 12, two case studies in Fig. 16, 17 in the appendix).
  • The average number of events generated by our safe-aware event sampler model in the training and testing set is 7.33 and 7.4. This reflects the high complexity and challenging nature of the videos in the guardrail task.
  • The average ratio of multi-labeled videos (i.e., videos flagged with multiple guardrail categories) is 24.7% in the training set and 28.57% in the testing set, further demonstrating the dataset's diversity and challenging nature.