Thanks for your constructive and insightful reviews. We carefully response to your concerns as follows.

Details

Thanks for your question. We provide the details regarding the training process in Section A.5.2 of our original paper. In the R-SFT stage, the total batch size is set to 192 and the initial learning rate is set to 5e-5. In the RL stage, the batch size for the actor model is set to 256 and the initial learning rate is set to 1e-6.

Length-aware Safety Reward

We design a safety reward to guide our guard model to finish two guardrail tasks, i.e., prompt harmfulness detection and response harmfulness detection. First, the model should output in a correct format to ensure the predicted results are extracted correctly. Then, based on the correct format, we calculate the correctness between the predicted results and the ground truth of these two tasks, and combine them linearly, as shown in Formulation (8) and (9) in the original paper.

To balance the performance and token efficiency, we incorporate the length of the reasoning process into the reward. The basic idea is that when the model fails to complete these guardrail tasks correctly, it is encouraged to improve its accuracy by scaling up the reasoning length, while remaining within a constraint: $r = \frac{-1+r_{\text{safety}}}{\min(l_{\text{norm}},\beta)^2}$, where $l_{\text{norm}}\in[0,1]$ is the normalized length of the reasoning $\mathcal{R}$, and $\beta$ is a cut-off hyper-parameter to alleviate over-thinking. Note that the numerator $r_{\text{safety}}$ is constrained to be non-positive, i.e., $r_{\text{safety}} \in [-1,0]$. Thus, when the model fails to complete all tasks correctly, i.e., $r_{\text{safety}} \in [-1, 0)$, it is encouraged to improve its accuracy by increasing the reasoning length, subject to the constraint $\beta$.

Basically, we design our length-aware reward to encourage the model to increase the reasoning length when it fails to solve the task correctly. However, this increase in reasoning length is constrained, preventing the model from continuously expanding the reasoning process.

Base Model & More Choice of Constraint Parameters

• We select Qwen2.5-VL-Instruct 3B and Qwen2.5-VL-Instruct 7B as our base models. We provide the reasons as follows. 1) During the process of this paper, Qwen2.5-VL models are the leading open-sourced vision-language models. We believe that we can achieve better performance based on them. 2) We merely select 3B and 7B models since we have limited computational resources. 3) For the guardrail tasks, we believe they are time-sensitive, which requires a small and efficient model.
• For a wider choice of the constraint parameters, we will conduct more experiments on models with different scales due to the limitation of the computational resources.

Response Length

First of all, we designed GuardReasoner-VL-Eco-7B to improve the token efficiency of the original GuardReasoner-VL-7B model. Therefore, it is both reasonable and expected that the reasoning length decreases with more training steps. The main reason for this is our implementation of a length-aware reward, as outlined in the Length-aware Safety Reward section. By adjusting the constraint parameter $\beta$, we can control the reasoning length, allowing us to strike a balance between performance and token efficiency. The more details of the hyper-parameter setting can be found in Section A.5.2 in the original paper.

Binary Classification

Conventional methods, such as LLaMA Guard, typically perform binary classification or fixed multi-class classification (for different harmful categories). In contrast, our model goes beyond binary classification by also outputting the reasoning process. This reasoning can serve as the justification for the final guardrail decision, including the identification of open-ended harmful categories. We think we have already solved this problem in our proposed model.

Dear Reviewer Wv3A,

We hope this message finds you well.

As the rebuttal period deadline is approaching, we would greatly appreciate it if you could kindly acknowledge and respond to our rebuttal for paper NeurIPS 12986. We are looking forward to receiving any further questions or suggestions for improvement. If your concerns have been resolved, could you raise your score to support our work?

Thank you very much for your support.

Kindest regards, Authors of NeurIPS 12986

Thanks for your constructive and insightful reviews. We carefully response to your concerns as follows.

Performance of GuardReasoner and GuardReasoner-VL

We admit that on some text-only benchmarks, our GuardReasoner-VL performs worse than GuardReasoner. However, we think it is reasonable since GuardReasoner is a text-only guard model while GuardReasoner-VL can handle multi-modal inputs, including text, image, and text-image modalities. In the future, we think this problem can be solved by scaling up the training data and the parameters of the VLMs.

Data Augmentation Strategy

We fully understand your concern. It is possible to lead to semantic ambiguity using data augmentations. However, the motivation of our designed data augmentation is to improve the difficulty of the training samples and improve the difficulty of the training process, thus enhancing the performance. As shown in Figure 5, we do observe that our proposed data augmentation strategy can improve the performance across various datasets and do not find that it will hinder the model training.

More Challenging Task

Our used benchmarks have already contained your mentioned kind of data, i.e., eliciting unsafe responses from models through safe input, a.k.a., jailbreak or adversarial attacks. Following your suggestions, we tested the performance on these benchmarks, i.e., MIS-hard (F1 Score), MSSBench Embodied Task (ACC), SIUO (F1 Score) [1-3]. The results are listed in the following table. We can find that our GuardReasoner-VL can achieve promising performance in these challenging benchmarks compared with the reported baselines in their original paper. We acknowledge these benchmarks are more challenging, and we will further improve the performance of our models in these benchmarks in the future. We will discuss these high-quality datasets and test our model in the final version of our paper.

[1] Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models 

[2] Multimodal situational safety 

[3]Cross-modality safety alignment

Single Inference

Yes, our GuardReasoner-VL can handle two tasks in a single inference. As shown in Equation (2), our guard model is designed to handle both prompt harmfulness detection task and response harmfulness detection task. For the case study, sorry for the misinformation. We merely demonstrate one task in one figure for the clarity.

Further Improvement

Yes, we believe both the performance and token efficiency can be improved by adjusting $\beta$. However, due to the limitation of the computational resources, we did not try too many values of this hyper-parameter. In the future, we can further improve the performance and the token efficiency of our model by tuning hyper-parameters.

Typo

Thanks for your reminder. We will fix it in the final version.

Thanks for your constructive and insightful reviews. We carefully response to your concerns as follows.

Data Types

We appreciate your point. However, the VLM guard models are specifically designed to safeguard VLMs, which process inputs containing various modalities, such as text, images, and text-image pairs. Therefore, training with text, image, and text-image data is essential and necessary.

Combinations

No, we do not use all possible combinations. Instead, we sample the combinations randomly. Please carefully review the details in Lines 630-632 of the original paper. In the future, we plan to develop more sophisticated sampling strategies to improve the model’s performance.

Imbalance

Thank you for raising this concern. We are aware that concatenating examples from different modalities could lead to potential imbalances, especially when dealing with varying image sizes or context lengths. One possible solution is to design sampling strategies that ensure a balance between different samples. We plan to incorporate this approach into our model in the future. Besides, although the potential imbalance problem exists, we observe that our data augmentation strategy works well and does improve the RL performance.

Efficiency of RL

We believe that a 2% performance improvement is not marginal, and the RL process is not inefficient, especially considering that we only use 12K training samples for RL compared to 123K training samples for SFT. In the future, we plan to scale up our prompt set and enhance its efficiency, which should further boost the performance of the RL process.

Hyper-parameter $B_s$

Thanks for your suggestion. We will consider adjusting the decay rate or implementing more gradual adjustments to ensure adequate exploration throughout the training process.

Human Involvement

Thank you for your question. Due to limitations in human resources, we are unable to involve the entire process of checking the reasoning paths. However, we conduct case studies to ensure the quality of the reasoning. Additionally, for the trained model, we perform human case studies, which are illustrated in Figure 12, Figure 13, and Figure 14. In the future, we plan to recruit more human labelers to further improve our model.

Best Result in Table 1

Thanks for your suggestion. In this table, we merely highlight the average performance for clarity. Following your suggestion, we will highlight all the best performances in the final version.

Performance of R-SFT (image)

Note that the variant refers to conducting R-SFT using only image data. Without the text data, the foundational abilities of the VLM guard model degrade rapidly. Please CAREFULLY read our paper, and we have already explained this issue in Lines 193-194 of the original paper.

Dear Reviewer 7Lz3,

We hope this message finds you well.

As the rebuttal period deadline is approaching, we would greatly appreciate it if you could kindly acknowledge and respond to our rebuttal for paper NeurIPS 12986. We are looking forward to receiving any further questions or suggestions for improvement. If your concerns have been resolved, could you raise your score to support our work?

Thank you very much for your support.

Kindest regards, Authors of NeurIPS 12986

Dear Reviewer 7Lz3,

Thanks for your quick response and insightful reviews. We respond to your further comments as follows.

You train on text, image, and text-image samples jointly, which raises the risk of the model over-relying on the textual modality. Could you provide analyses showing that joint training does not introduce a bias toward text?

Thanks for your question.

• For your concerns, we have already conducted experiments by separating the different modalities of the training data (see Figure 5). From these experimental results, we can find that R-SFT (which trains the model with text, image, and text-image) can achieve better performance than R-SFT (text). This suggests that introducing the image modality can enhance performance, demonstrating that the model's performance doesn't over-rely on the textual modality. We would like to add this analysis to the final version of our paper.

• Besides, we also analyze the performance of our model on data with different modalities. For example, in Table 1, our model can achieve promising performance on HarmImageTest (which only contains images) and SPA-VL-Eval (which only contains image-text pairs). These results also verify that our model has the generalization ability to image and text-image modalities, and our model doesn't have the risk of over-relying on the textual modality.

Following your suggestions, we added these discussions and highlighted them in red in the revised paper. Please refer to the original anonymous link in our paper.

Could you also show that the imbalance does not degrade the overall data quality?

Thanks. We admit the imbalance could degrade the overall data quality. However, the overall performance improvement shows the effectiveness of our overall data. We think this problem can be solved by designing a new sampling strategy, i.e., when the imbalance happens, we discard the corresponding samples. Besides, we carefully checked our data augmentation and didn't find any imbalance problem. The cases can be found in Figure 4 of the original paper.

Following your suggestions, we have added these discussions and highlighted them in red in the revised paper. Please refer to the original anonymous link in our paper.

Overall, we thank you for your constructive and insightful reviews and comments. We are looking forward to receiving any further questions or suggestions for improvement. If your concerns have been resolved, could you raise your score to support our work?

Dear Reviewer 7Lz3,

We have updated our response. Concretely, we add more analyses on text data over-relying and the data imbalance. Besides, we have added them to our revised paper. Please refer to the original anonymous link in our paper.

We appreciate your constructive and insightful reviews and comments. We are looking forward to receiving any further questions or suggestions for improvement.

Best Regards, Authors

Thanks for your constructive and insightful reviews. We carefully response to your concerns as follows.

GPT-4o

• Legal Issue. We agree with your point. This learning process is called distillation. Actually, various famous models are using this kind of technology, e.g., DeepSeek and Qwen. However, our model is only for research and not for commercial use, thus avoiding the legal issue. We just study a new paradigm for training a reasoning-based guardrail model. If the companies plan to train a reasoning-based model, they can use our techniques and may label the reasoning data by themselves.

• Prompting GPT-4o Directly. Actually, we have tried prompt GPT-4o to conduct the guardrail tasks directly (note that it is different from prompt GPT-4o to generate the intermediate reasoning processes ). However, when we prompt GPT-4o directly to conduct harmfulness detection tasks, we got an unpromising performance since it will always reject our requests. Differently, when we just prompt GPT-4o to generate the intermediate reasoning processes for the guardrail tasks, it works well and outputs some high-quality reasoning data. We find this interesting point and may attribute it to one kind of jailbreak attack. We will add it to the final version of our paper.

Chain-of-Thought

We agree with your point. This is a kind of CoT techniques, which is first proposed in 2022. However, our method is one kind of learning to reason techniques, i.e., teaching the model to learn to reason, which is brought by o1 model in 2024. Besides, our proposed GuardReasoner-VL is the first reasoning-based VLM guardrail model.

Baselines

As shown in Table 2, it includes not only small models but also advanced moderation APIs, such as the OpenAI Moderation API and the Azure Content Safety API. For the performance of directly prompting GPT-4o, we explain it in the GPT-4o section. Basically, when we directly prompting GPT-4o to accomplish the guardrail task, it always denies our requests, thus leading to an unpromising performance.

Human Evaluation

Actually, we have already conducted some human evaluation in our original paper. As shown in Figure 12, Figure 13, and Figure 14, we find that our proposed reasoning-based guardrail model can achieve better performance, and the reasoning process does help to improve the accuracy and the interpretability.

Intuition

We think that reasoning traces can help the model better analyze both the user's input and the model's response. By providing a step-by-step trace of the reasoning process, the model can identify key information, make more informed decisions, and improve its overall performance. This transparency also helps in refining the decision-making process and making the model's behavior more interpretable. Also, this reasoning process can be regarded as one kind of test-time scaling, which can improve the model's performance. Some intuition can be found in our case studies. And we will add these insights to our final version.

Dear Reviewer VpvK,

We hope this message finds you well.

As the rebuttal period deadline is approaching, we would greatly appreciate it if you could kindly acknowledge and respond to our rebuttal for paper NeurIPS 12986. We are looking forward to receiving any further questions or suggestions for improvement. If your concerns have been resolved, could you raise your score to support our work?

Thank you very much for your support.

Kindest regards, Authors of NeurIPS 12986

Dear Reviewer VpvK,

We hope this message finds you well.

As the rebuttal period deadline is fast approaching (in less than 2 days), we would kindly appreciate it if you could acknowledge and respond to our rebuttal for paper NeurIPS 12986. We have responded to and solved your concerns. We are eager to address any further questions or suggestions you may have for improvement. If your concerns have been resolved, we would be grateful if you could consider raising your score or your confidence in support of our work.

Thank you very much for your time and support.

Kindest regards, Authors of NeurIPS 12986