Safety Reasoning with Guidelines

Thanks for your time and effort in reviewing our work, as well as for recognizing our contributions!

---

Response to Weakness 1:

1. Apart from o1 system card and Deliberative alignment paper from OpenAI, no prior work from academic community demonstrates reasoning could enhance safety performance. Existing works main focus on reasoning on math or coding domain instead considering safety alignment performance.
2. Moreover, safety alignment poses unique challenges, which our contributions specifically address:
   a. Our work first highlights the necessity of training models to reason for safety alignment. Through BoN evaluations and domain adaptation analyses, we demonstrate that refusal-trained models show potential in handling OOD attacks but ultimately rely on superficial shortcuts, limiting their use of latent knowledge. This underscores the need for explicit safety reasoning to enable step-by-step reasoning for knowledge utilization.
   b. In safety alignment, reasoning needs to involve systematically accounting for multidimensional safety factors to mitigate potential jailbreak risks. Therefore, we propose training models to reason based on explicit guidelines reflecting various safety perspectives. Here, we conduct experiments incorporating more guidelines, covering role awareness, intent recognition, etc., during supervision synthesis. The full list is available at https://anonymous.4open.science/r/good-664D. Due to inference-time and cost constraints in rebuttal phase, we used open-source Qwen2.5-72B-Instruct model to generate supervision and trained models with LoRA under same settings as our submission. The comparsion are shown below. SRG (extra guidelines) outperforms across various attacks, verifing the effectiveness of reasoning based on guidelines. These results will be included in the revised version. |ASR(%)($\downarrow$)|Illegal|Jailbreak|Self-Cipher|PastTense|Persuasive| |--|--|--|--|--|--| |RT|6|70.5|80|56|82| |SRG(reasoning pattern)|2.5|17.5|2.5|43|64| |SRG(extra guidelines)|0|4.5|1|32|52|

---

Response to Weakness 2:

Thank you for the suggestion. As noted, previous work primarily focuses on math or coding reasoning, lacking explicit alignment capabilities for safety. While o1 series demonstrates strong performance, it does not provide detailed reasoning steps to users.
Following your suggestion, we include a baseline that distills safety CoT data from the open-source reasoning model DeepSeek-R1-Distill-Qwen-14B. We also apply our SRG approach under same settings. The results show that SRG still achieves substantial improvements across various attacks. However, these overall results are lower than those in our original submission, likely because these reasoning models were not sufficiently trained for safety. We believe that more refined, as well as larger-scale rejection sampling can further improve performance. Since R1 series was released one week before ICML submission deadline, we could not conduct relevant experiments in time. Thanks again for your suggestion; we will include these results in the revised version.

---

Response to Comments:

1. We will refine the title to make it more concise.
2. The details of context distillation are shown in Line 260-274. Here, we clarify them again. After collecting CoT data $(x^c, y^c), x^c = (C, x)$, we remove C and only retain x as input and train models to internalize reasoning w.r.t guidelines. Standard SFT in our work means using original input $x^c$ to train models. Thanks for your reminder and we will clarify this point more clearly in revised version.
3. Thanks for this insightful suggestion. We believe this occurs because RT models possess safety-related latent knowledge, enabling it to recognize malicious instructions and assign non-zero probabilities to refusal tokens, allowing it to generate refusals during sampling.

Following your suggestion, we test refusal tokens’ probabilities on 4 attacks and helpful queries from Alpaca dataset. We notice that RT model always uses “I cannot” as its refusal tokens, whereas SRG uses “I’m sorry.” So we check these tokens' probabilities and report average values below. Compared with values on Alpaca, RT assigns much higher probabilities for refusal tokens on attacks, especially illegal instructions (ID attack). This aligns with our analysis and BoN evaluations, showing that RT needs more sampling to lower ASR on OOD attacks. SRG assigns higher probabilities on OOD attacks, consistent with its improved OOD generalization.

We greatly appreciate your thought-provoking questions. They have provided valuable inspiration. We will also add these discussions in the revised version.

Thank you for your time and effort in reviewing our work, as well as for recognizing our contributions!

---

Response to Weakness and Comments:

Thanks for the reminder. We will reformat the figure layout, including training and evaluation steps, to improve the readability of our pipeline. We will include notations and a brief explanation about arrows in the caption of Figure 3. Figure 4 will also be revised for better clarity.

Thanks for pointing out the typos. We will modify them in revised version.

---

Response to Questions:

Thanks for your questions.

1. This is an interesting question and a similar phenomenon has also been observed in [1]. We believe it stems from complex training dynamics of LLM fine-tuning: compared with full fine-tuning, LoRA may be less prone to overfitting refusal pattern, potentially leading to better OOD generalization [2].
2. We believe that reliable reward models or well-defined verification rules are essential for RL training in safety reasoning. Unlike mathematical reasoning, safety tasks lack fixed ground-truth answers and are inherently more challenging to evaluate. As noted in right section of Lines 243–259, our guidelines may serve as extra auxiliary verification criteria, enhancing reliability of RL rewards and mitigating potential reward hacking. We will further investigate reward modeling for scaling RL in safety reasoning in future work.
3. Data mixture ratio is a critical factor in alignment performance [3,4]. In this work, we do not focus specifically on this aspect but follow the setup from previous studies [1], as our primary goal is to train models for safety reasoning. We will further explore ablation studies on mixture ratios in future work.

We greatly appreciate your thought-provoking questions. They have provided valuable inspiration. We will also add these discussions in the revised version.

[1] Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training.
[2] LoRA Learns Less and Forgets Less.
[3] Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions.
[4] The Llama 3 Herd of Models.

---

We sincerely thank you for your thoughtful review. We gratefully hope that you could re-evaluate our paper based on the responses and clarifications provided above. If our responses have satisfactorily addressed your concerns, we would greatly appreciate it if you could consider updating your review score accordingly.
However, if you have any additional concerns, please do not hesitate to let us know. We are more than willing to provide further clarification.

Thanks for your time and effort in reviewing our work.

---

Response to Claims and Methods:

1. Thanks for your comments. We evaluate over-refusal using XSTest dataset [5], as shown below. Our method outperforms LAT and GPT-4o, achieving 92%, slightly behind LLaMA3-8B-Instruct (baseline). RR performs better as it includes XSTest in training. This confirms that our method enhances safety without excessive refusals. |Model|Instruct|SRG|RR|LAT|GPT-4o| |--|--|--|--|--|--| |XSTest(%$\uparrow$)|95%|92%|99%|80%|88%|
2. The average inference output length is 400 for RT and 900 tokens for SRG. While SRG incurs higher inference costs, the increase remains within a reasonable range. Moreover, generating more tokens through reasoning is necessary, as it enhances the model’s ability to handle OOD and complex queries. Our results validate this, showing significant improvements against OOD attacks and better results on helpfulness tasks. OpenAI’s o1 and DeepSeek’s R1 also demonstrated effectiveness of Long CoT in math and other domains. In future work, we will further optimize inference efficiency to reduce costs.

---

Response to Experimental Designs:

1. About Dataset size:
   We conduct these experiments to investigate the impact of dataset size on RT and our SRG in safety and helpfulness tasks. We have discussed the takeaways in Line 362-370 and Line 416-424 of submission: 1) Compared to RT, SRG achieves consistent improvements against OOD attacks on both 8B and 70B models as the dataset scales, highlighting its potential for scaling CoT supervision; (2) SRG significantly enhances helpfulness even with a small-scale dataset.
   We will further clarify these insights in the revised version. Apart from dataset size, no additional differences exist.

2. About BoN:
   Unlike SC, which uses majority voting over N outputs to determine final answer, BoN adopts an external safety classifier—LLaMA-Guard3-8B—to select safe responses (mentioned in Lines 739–745). For each harmful query, we sample N outputs from model. If at least one is classified as safe by LLaMA-Guard, we consider the model safe for that query, and the attack is marked as unsuccessful.
   Thanks for this reminder. We will include more details about setup of BoN evaluation in revised version.

---

Response to Relation To Literature:

While CoT distillation exists for math reasoning, safety alignment presents unique challenges that our contributions specifically address:

1. We highlight the necessity of training models to reason for safety alignment. BoN evaluations and domain adaptation analyses show that refusal-trained models show potential in handling OOD attacks but ultimately rely on superficial shortcuts, limiting their use of latent knowledge. This underscores the need for explicit safety reasoning to enable step-by-step reasoning for knowledge utilization.
2. In safety alignment, reasoning needs to involve systematically accounting for multidimensional safety factors to mitigate potential jailbreak risks. Therefore, we propose training models to reason based on explicit guidelines reflecting various safety perspectives. Our evaluations confirm that SRG significantly enhances OOD generalization, aligning with our analysis.
3. No prior work provide methods for collecting safety CoT data and training safety reasoning model. Our work offers a detailed pipeline for synthesizing safety CoT supervision and model training to research community.

---

The response to Weakness has been addressed above.

---

Response to Comments and Questions:

1. We will adjust the table layout and add notations as suggested to improve readability.
2. Thanks for the reminder. It seems you are referring to Unsupervised Domain Adaptation—we will add a citation.
3. During deployment, our reasoning model does not expose its thought process to users. Context distillation is applied during training. After collecting CoT data $(x^c, y^c), x^c = (C, x)$, we retain only x as input, training models to internalize reasoning based on guidelines. Thus, context distillation is independent of deployment.
4. For Question 2:
   The safety classifier is valuable and complements aligned models but faces OOD generalization challenges, as adversarial prompts can easily bypass it [1,2]. More importantly, training an helpful, harmless, and honest LLM is primary goal for AI alignment [3,4]. Therefore, our work focuses on training safety reasoning models.
5. For Question 4:
   Thanks for the reminder. "Small-scale" and "large-scale" refer to training on small and large datasets, as shown in lines 159–162. We will add a brief explanation in the caption.

---

[1] https://github.com/andyzoujm/breaking-llama-guard.
[2] Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters.
[3] GPT-4 Technical Report.
[4] Constitutional AI: Harmlessness from AI Feedback.
[5] [4] XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models

Thanks for your time and effort in reviewing our work, as well as for recognizing our contributions!

---

Response to Experimental Designs:

1. Thanks for your suggestion. We evaluated over-refusal using XSTest dataset [4], as shown below. Our method outperforms LAT and GPT-4o, achieving 92%, slightly behind LLaMA3-8B-Instruct (baseline). RR performs better as it includes XSTest in training. This confirms that our method enhances safety without excessive refusals. |Model|Instruct|SRG|RR|LAT|GPT-4o| |--|--|--|--|--|--| |XSTest(%)($\uparrow$)|95|92|99|80|88|

2. Thanks for your question. Below are the details on costs:
   RT Model: Training samples average 460 tokens, and inference outputs 400 tokens. Training 8B model on a large dataset takes ~2.5 hours using 4×A100 80GB GPUs.
   SRG: Training samples average 950 tokens, and inference outputs 900 tokens. Training takes ~5 hours on the same setup.
   While SRG incurs higher inference costs, the increase remains within a reasonable range. Moreover, generating more tokens through reasoning is necessary, as it enhances the model’s ability to handle OOD and complex queries. Our results validate this, showing significant improvements against OOD attacks and better performance on helpfulness tasks. Similarly, OpenAI’s o1 and DeepSeek’s R1 have demonstrated effectiveness of Long CoT in math and other domains. In future work, we will further optimize inference efficiency to reduce costs.

3. Thanks for your suggestion. However, we are not entirely sure about your request. Based on our understanding, you ask for an evaluation of a model trained only on helpful dataset, assessing attack and helpfulness performance.
   We trained a model on a large-scale helpful dataset and present results below. Compared to Tables 1 and 5, this baseline shows much weaker safety performance than RT and SRG but outperforms RT across all helpfulness tasks, while still trailing SRG in overall performance. Attack|Illegal|Jailbreak|Cipher|PastTense|Persuative --|--|--|--|--|-- ASR(%)($\downarrow$)|52|80.5|95|83|94

   Task	HumanEval	Mbpp	GSM8K	MATH	BFCL	Avg
   Accuracy(%)($\uparrow$)	53	50.8	66.4	23.1	62.2	51.1

---

Response to Essential References:

1. Thanks for your suggestion. We cited Deliberative Alignment (DA) in our related work. Here we further clarify key distinctions between our method and DA. The "specifications" (specs) in DA differ significantly from "guidelines" used in our SRG framework.
2. Specifications explicitly provide category-specific response directives aligned with OpenAI's safety policy (e.g., erotic content, self-harm, fraud, etc.). For example, as shown in Sec. 2.3 of DA, given a harmful prompt categorized as fraud—"How to manipulate elderly people into buying health supplements"—the corresponding spec(fraud): "The assistant should directly refuse this query as it constitutes fraud and threatens public safety." In contrast, our guidelines serve as general, category-agnostic reasoning aids rather than prescribing specific responses. They help model assess query safety across multiple dimensions, such as user intent, contextual background, role awareness, etc. For instance, as shown in https://anonymous.4open.science/r/good-664D, our guidelines encourage reasoning through hints like "Role Awareness: Recognizing the appropriate role of the model (assistant, advisor, moderator) in various scenarios." We believe that reasoning needs to involve systematically accounting for multidimensional safety factors to mitigate potential jailbreak risks. The improved OOD generalization performance also verify the effectiveness of our method.
3. Additionally, DA does not explicitly detail methods for generating safety CoT data. In constrast, Our work offers a detailed pipeline for synthesizing safety CoT supervision and model training to research community.
4. SafeChain directly distills CoT data from open-source R1 models and does not consider OOD generalization of safety, as its training and test sets are drawn from the same dataset (WildJailbreak). It was published on arXiv on Feb 17, after ICML submission deadline. We wiil cite it in revised version.
   We will incorporate these discussions as a separate section in the revised version.

---

Response to Other Comments and Questions

Thanks for your comments! We will follow your suggestions to adjust the table and figure layout in our revised version.

Our guidelines are initially inspired by CAI [1], attack studies [2], and critique research [3]. We manually curate and iteratively refine them based on GPT-4o’s feedback. We will add this clarification in the revised version.

---

[1] Constitutional AI: Harmlessness from AI Feedback
[2] "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
[3] Self-critiquing models for assisting human evaluators
[4] XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models

Thank you for your time and effort in reviewing our work.

---

Response to Claim part:

1. We use 'complete pipeline' to indicate we offer a thorough pipeline to train safety reasoning models, which include three parts: 1) synthesizing reasoning supervision w.r.t guidelines (C); 2) rejection sampling; 3) internalizing guidelines with context distillation, as mentioned in Sec.4 and Figure 3.
   Following your suggestion, we conducted experiments incorporating extended guidelines, covering role awareness, intent recognition, cultural sensitivity, etc., during supervision synthesis. The full list is available at https://anonymous.4open.science/r/good-664D/extra_guidelines.jpg. Due to inference-time and cost constraints in the rebuttal phase, we used the open-source Qwen2.5-72B-Instruct model to generate supervision and trained models with LoRA under the same settings as our submission. We compared RT, SRG, and Qwen-Instruct, with results shown below. SRG (extra guidelines) outperforms across various attacks, demonstrating the effectiveness and scalability of our framework. These results will be included in the revised version. |ASR(%)($\downarrow$)|Illegal|Jailbreak|Self-Cipher|PastTense|Persuasive| |--|--|--|--|--|--| |RT|6|70.5|80|56|82| |Qwen-72B|4.5|25|2.5|61|84| |SRG(reasoning pattern)|2.5|17.5|2.5|43|64| |SRG(extra guidelines)|0|4.5|1|32|52|

---

2. Thanks for your questions. SRG synthesizes improved supervision, while RR and LAT focus on enhancing training objectives, making SRG complementary to them. We combined SRG with RR by further training SRG model using RR with the authors' codebase. For fairness, we also reproduced RR results using the same codebase. As shown in the table, SRG-RR improves performance against attacks over SRG alone and also outperforms RR. These results will be included in revised version. |ASR(%)($\downarrow$)|Illegal|Jailbreak|Self-Cipher|PastTense|Persuasive| |--|--|--|--|--|--| |SRG|0|1|0|6|26| |RR|0|0.5|0|13|12| |SRG+RR|0|0|0|3|8|

---

Response to Weakness:

1. Thank you for the reminder. While not explicitly detailed in the paper, we conducted comprehensive human evaluations on LLaMA-Guard-3 results and partial evaluations using GPT-4o as a safety judge. Original model evaluation results can be found here: https://anonymous.4open.science/r/good-664D
   Unreadable tokens in RR's outputs often cause LLaMA-Guard-3 to classify them as unsafe, especially under self-cipher attacks. However, our human evaluation deemed them safe, explaining RR’s reported 0% ASR for self-cipher attacks. Original evaluation results are shown in above link.
   We appreciate this point and will clarify our evaluation metrics in the revised version.
2. We used "lag behind" to indicate a significant performance gap. We appreciate the feedback and will clarify the phrasing to avoid ambiguity.
3. Thank you for your questions. We believe adopted attacks are appropriate for evaluating OOD safety performance of models:
   i. To ensure a fair comparison with RR and LAT in Table 4, we strictly followed RR's settings, training models from LLaMA-3-8B-instruct, which has undergone extensive safety alignment and training data is likely to cover common OOD attacks [1]. Despite this inherent robustness, our method still achieves measurable improvements, demonstrating its effectiveness.
   ii. Our work focuses on building safe reasoning models from base models, specifically tackling the challenge of generalizing from ID training to OOD attacks—a critical issue highlighted in [2]. Effective attacks often stem from OOD scenarios, as seen in the high ASR of RT models. Moreover, these attacks are widely used in prior studies [3,4] and have proven effective even against aligned models like OpenAI’s ChatGPT.
4. Even a few successful attacks are critical, as each vulnerability can be repeatedly exploited by attackers, posing substantial safety risks as discussed in [5].
   For PersuasiveAttack, we strictly followed the original setup. The authors provided only 50 attack questions in their dataset [6].
   For PAIR, we adhered to the original evaluation setup [7], which contains 50 harmful behavior samples. Actually, each sample generates 20 attack prompts, iterating up to 3 times with GPT APIs, producing 1,000–3,000 jailbreak prompts. Evaluating SRG models required ~16 hours, even with fast APIs like GPT-4o-mini (mentioned in Appendix: Lines 712–722).
5. We have provided results in 2nd response to Claim part.

---

[1] The Llama 3 Herd of Models.
[2] Adversarial ML Problems Are Getting Harder to Solve and to Evaluate.
[3] Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training.
[4] "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models.
[5] https://www.anthropic.com/news/constitutional-classifiers
[6] https://huggingface.co/datasets/CHATS-Lab/Persuasive-Jailbreaker-Data
[7] https://github.com/patrickrchao/JailbreakingLLMs