RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards

Comment 1: Straightfoward reward components with minimal adaptation

"At its core, RSafe applies reinforcement learning with two straightforward reward components—format compliance and classification accuracy. While effective, this combination follows established training patterns with minimal adaptation.

Thank you for raising this point. RSafe's 'minimal' reward design is not simplistic by oversight but by thoughtful choice: it provides a direct, noise-free approach to optimizing safety moderation objectives, avoiding reliance on intermediate reward models that could introduce noise or the pitfalls of complex reward designs. More specifically, Reinforcement learning (RL) algorithms are :

1. Highly sensitive to reward noise [1]. Adopting verifiable ground-truth labels avoids extra stochasticity that would otherwise be introduced by a learned reward model or external classifier.;
2. Prone to reward hacking, where the policy model can exploit flaws in the reward function rather than learning the intended behavior [2]. RSafe's concise verifiable reward surface leaves fewer unintended optima.

We outline several potential extensions below and explain why we approach them cautiously, ultimately returning to RSafe's straightforward design.

Comment 2: Reliance on the capabilities of the backbone model

"The guided reasoning and RL are only conducted on Qwen2.5-7B."

Thank you for raising this point. We have added experiments using Qwen2.5-3B and Llama3.1-8B as alternative backbones, with updated results for effectiveness and robustness shown in Table 1 and Table 2. The results confirm that the choice of backbone influences performance: Qwen2.5-3B performs worse than Qwen2.5-7B, while Llama3.1-8B performs slightly worse on effectiveness but achieves higher robustness. These results suggest that RSafe can better leverage stronger backbone models to enhance safety through more robust reasoning capabilities.

Table 1. Performance comparison on safety moderation benchmarks across different backbone models.

Table 2. Robustness comparison on WildGuardTest across different backbone models.

Comment 3: Disclosure of training details

"The mechanism by which dynamic safety categories are integrated into training is described only at a high level."

Thank you for raising this issue. During training, we used a default instruction prompt set providing in-distribution data to teach the model context-aware reasoning based on given safety policies. At inference, we support dynamic safety categories, allowing users to specify their required safety policies based on their needs.

To improve reproducibility and address practical overhead, we provide detailed training data scale and compute usage in Appendix C.3. In total, we curated a training dataset of approximately 12,000 examples, balanced across harmful and unharmful samples from five safety-related corpora, covering both prompt-level and response-level harmfulness detection tasks across diverse safety scenarios.

For computational resources, we used the VERL codebase on 4×A100 80GB GPUs with a batch size of 128 and a maximum sequence length of 2048 tokens. Each run was trained for 3 epochs with rollout as 4, taking approximately 3–4 hours.

Comment 4: Missing / mis-specified safety taxonomies

"The adaptive mode presumes users can accurately define relevant safety taxonomies; in practice, overlooking critical policy categories (e.g., emergent threats) may degrade performance."

Thanks for bringing up this problem. When users encounter novel safety situations and cannot supply a clear policy, RSafe provides an option to omit the safety taxonomy while still providing strong moderation. Table 4 presents this configuration as “RSafe (w/o category specification).”, where it outperforms its baselines and the default RSafe version. RSafe can draws on its internal safety self-awareness [5] and the context-aware reasoning capability acquired during training to provide flexible and accurate decisions, instead of trying to fit new hazards into specified safety categories that may be ill-suited. These results demonstrate that RSafe remains both robust and protective even when users cannot specify a taxonomy and the scenario lies outside all predefined categories.

Table 4. RSafe's performance without safety category specification compared to its adaptive mode and other baselines on out-of-domain benchmarks featuring novel safety categories.

References

[1] Liang, Xize, et al. "ROPO: Robust Preference Optimization for Large Language Models." (ICML 2024).

[2] Fu, Jiayi, et al. "Reward shaping to mitigate reward hacking in rlhf." (2025).

[3] Mroueh, Youssef. "Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification." (2025).

[4] Lin, Yuping, et al. "Towards understanding jailbreak attacks in llms: A representation space analysis." (2024).

Thanks for your time and your comment, regarding your concerns:

1. The concern that the lack of supervision over intermediate steps might introduce instability does not hold in our setting. Through manual inspection, we repeatedly confirm the consistency between the reasoning path and the final prediction. Moreover, eliminating intermediate supervision mitigates the risks of label noise and reduces annotation burden, ultimately enhancing the robustness of our framework.

2. RSafe, trained on Qwen2.5-7B with approximately 10k examples, achieves competitive overall effectiveness compared to GuardReasoner, which is trained on Llama3.1-8B with 127k examples. For a controlled comparison, we conducted experiments using the same backbone model and training datasets, with the only difference being the training pipeline. The results, presented in Table 1 and Table 2, demonstrate that RSafe's superior performance.

Table 1. Performance comparison on safety moderation benchmarks across different methods.

Table 2. Robustness comparison on WildGuardTest across different methods.

3. While providing accurate and fine-grained safety categories yields the best results, RSafe remains robust even without such specifications, according to results shown in Table 4 in the rebuttal response. Deliberately specifying incorrect or contradictory categories falls outside the scope of our intended use and is thus not addressed in this work.

Comment 1: Robustness on edge cases / adaptiveness to highly specific safety rules

"One evaluation that would be interesting would be to see how RSafe handle entirely idiosyncratic, edge-case, or otherwise highly specific user‑defined safety rules, particularly when compared with GPT‑4o or Claude.""

Thank you for raising this interesting question. We interpret your suggestion as encompassing two aspects: (1) robustness evaluation under edge-case adversarial prompts, such as those from red-teaming scenarios, and (2) adaptivity evaluation under highly specific user-defined safety rules. We conducted two separate experiments accordingly.

For the first aspect, we focused on edge-case prompts from two red-teaming datasets: JailbreakBench [1] and JailJudge [2]. These datasets include jailbreak attacks (e.g., PAIR [3], GPTFuzzer [4]) and deceptive strategies such as role-playing and hypothetical scenarios, as well as prompts based on real-world risks. To ensure the selected prompts represent genuinely challenging safety risks that are not trivially detectable, we filtered out prompts that GPT-4o refused outright with generic rejections like "I'm sorry, I can't help with...". After filtering, 17 prompts from JailbreakBench and 432 from JailJudge remained.

We then used RSafe and GPT-4o with identical instruction prompts to perform safety moderation. As shown in Table 1, RSafe performed slightly better than GPT-4o on JailJudge but worse on JailbreakBench, likely due to the small sample size in the latter making the difference appear more pronounced. While RSafe's accuracy in detecting safety risks drops compared to its ~0.9 average on in-domain general safety benchmarks (calculated from Table 1 in the original paper), it maintains strong protective capability and matches GPT-4o when used as LLM-as-judge, demonstrating robustness among edge cases.

Table 1. Robustness on edge cases: RSafe vs. GPT-4o.

For the second experiment evaluating adaptiveness, we follow the setup from Aegis 2.0 [5] to synthetically generate prompts that either violate or comply with fine-grained guidelines across three domains: Financial Advice, Legal Advice, and Medical Advice. We construct dataset with 40 prompts per category, balanced between violations and compliant cases.

As shown in Table 2, both RSafe-adaptive and GPT-4o achieve near-perfect accuracy. This result is expected: the policies are clearly defined, and the synthetic prompts are tightly aligned with the policy framing, making this task primarily a test of instruction-following rather than nuanced reasoning. By contrast, the adversarial prompts in Table 1, drawn from red-teaming attacks, are significantly more challenging and more likely to bypass safety defenses. This highlights the importance of focusing on adversarial/edge-case examples when evaluating robustness under highly specific, user-defined safety scenarios.

Table 2. Adaptiveness on highly user-specific rules: RSafe vs. GPT-4o.

Comment 2: Adaptiveness under safety category exclusion

"How does RSafe work when certain safety metrics are excluded""

Thank you for bringing this up. RSafe-adaptive allows users to specify safety policies during inference, enabling the exclusion of certain policies when specific risks are not relevant. Based on your suggestion, we added evaluations to test RSafe's ability to exclude safety policies.

We used three benchmarks with annotated safety taxonomies: Sorry-Bench, HarmBench, and WildGuardTest. We evaluated two settings: (1) RSafe-adaptive with full taxonomy adherence, and (2) RSafe-adaptive (w/ removal) that exclude specified categories. For example, in Sorry-Bench, we removed five subcategories under Hate Speech Generation, leaving 40 subcategories, then examined performance on both removed and remaining categories. GPT-4o was included for comparison using identical prompts.

As shown in Table 3, excluding categories had no significant impact on remaining category performance. However, both RSafe and GPT-4o showed limited ability to remove refusals in excluded categories.

Investigation revealed that prompts failing to be excluded often violated multiple taxonomies simultaneously. For instance, prompts annotated as hate speech frequently triggered other safety categories due to policy overlaps. Even after removing Hate Speech Generation, RSafe continued flagging these prompts under different taxonomies. Manual inspection across all benchmarks confirmed this overlap was consistently the cause. Successfully excluded prompts violated only the removed category.

The result demonstrates RSafe's policy exclusion capability. However, safety taxonomies are not disjoint, and prompts often violate multiple categories, making complete exclusion non-trivial.

Table 3. Adaptiveness comparison on exclusion tasks for RSafe and GPT-4o.

Comment 3: Expand robustness evaluation

"Is there possibility to expand the testing of robustness beyond WildGuardTest (Section 4.2), toward evals that demonstrate that it can follow instructions with respect to novel safety criteria?"

Thank you for raising this point. We expanded the original Section 4.2 experiments on WildGuardTest to validate RSafe's adaptivity to novel safety criteria by adding two medical ethics benchmarks: MedSafetyBench and CARES, with CARES including both base and adversarial versions. As shown in Table 3, OpenAI Moderation, ShieldGemma-9B, LlamaGuard3-8B, and RSafe all exhibit significant accuracy drops on these datasets compared to their average performance on general safety benchmarks (results from Table 1 in the original paper). This indicates that MedSafetyBench and CARES present out-of-distribution safety challenges beyond all models' training domains.

As shown in Table 3 below, RSafe outperforms baseline models on both datasets. When guided by specific medical principles at inference (RSafe-adaptive), performance improves notably. Compared to LLM-as-judge (GPT-4o with corresponding ethical guidelines), RSafe-adaptive achieves comparable performance, demonstrating strong adaptiveness and reliability as a safety guardrail model.

Table 3. Adaptiveness comparison of RSafe and baseline models on novel safety topics.

References

[1] Chao, Patrick, et al. "Jailbreakbench: An open robustness benchmark for jailbreaking large language models." (NIPS 2024).

[2] Liu, Fan, et al. "Jailjudge: A comprehensive jailbreak judge benchmark with multi-agent enhanced explanation evaluation framework." (2024).

[3] Chao, Patrick, et al. "Jailbreaking black box large language models in twenty queries." (2023).

[4] Yu, Jiahao, et al. "Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts." (2023).

[5] Ghosh, Shaona, et al. "Aegis2.0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails." (2025).

Comment 1: Comparison between DPO and GRPO

"The proposed formulation is closely related to the GuardReasoner framework. The principal difference is that GuardReasoner relies on DPO, whereas the present work adopts GRPO with verifiable rewards such as accuracy and format compliance. This change, while sensible, may appear incremental and therefore limits the novelty of the contribution."

Thank you for your thoughtful comment. While the transition from DPO to GRPO may initially seem like a natural extension, from pairwise preference learning to verifiable reward optimization, and merely a change in the loss function, GRPO in fact introduces several essential capabilities that go beyond this surface-level shift. Specifically, GRPO enables more effective alignment in safety moderation tasks by leveraging (1) verifiable supervision, (2) multi-objective reward integration, and (3) on-policy optimization. These properties make GRPO particularly well-suited for safety moderations, as explained in details below.

Comment 2: Demonstrate adaptiveness to truly novel safety topics

"In the robustness experiment on unseen safety categories, the “Criminal Planning / Confessions” topic can overlap with categories already covered by Llama-Guard (violent crime) and ShieldGemma (dangerous content). To better demonstrate adaptiveness to truly novel topics, I suggest (i) choosing a domain unrelated to safety—e.g., a keyword-mentioning or mathematics discussion task—and (ii) adding an LLM-as-judge baseline (such as GPT-4o) to evaluate safeguard reliability when judgment is delegated to an external model."

Thanks for raising this point. The initial adaptiveness evaluation dataset WildGuardTest may overlap with the training data of some baseline guard models, potentially favoring them. To mitigate this, we first evaluated on GSM8K, a math dataset unrelated to safety. As expected, all models, including RSafe and its baselines, correctly identified the content as harmless.

To better assess adaptiveness to novel safety topics, we turned to two medical ethics benchmarks: MedSafetyBench [7] and CARES [8], the latter including both base and adversarial versions. As shown in Table 1, OpenAI Moderation, ShieldGemma-9B, LlamaGuard3-8B, and RSafe all show significant drops in accuracy on these datasets compared to their average performance on general safety benchmarks (from Table 1 in original paper). This indicates that MedSafetyBench and CARES present out-of-distribution safety challenges beyond the training domains of all models.

RSafe outperforms its baselines on both datasets. When guided by specific medical principles at inference (RSafe-adaptive), its performance improves notably. Compared to LLM-as-judge (GPT-4o provided with corresponding ethical guidelines), RSafe-adaptive achieves comparable performance, demonstrating strong adaptiveness and reliability as a safety guardrail model.

Table 1. Adaptiveness comparison of RSafe and baseline models on novel safety topics.

References

[1] Liu, Yue, et al. "Guardreasoner: Towards reasoning-based llm safeguards." (2025).

[2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." (NIPS 2023).

[3] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." (2025).

[4] Xu, Shusheng, et al. "Is dpo superior to ppo for llm alignment? a comprehensive study." (ICML 2024).

[5] Wu, Junkang, et al. "Towards robust alignment of language models: Distributionally robustifying direct preference optimization." (ICLR 2025)

[6] Wang, Zhichao, et al. "A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more." (2024).

[7] Han, Tessa, et al. "Medsafetybench: Evaluating and improving the medical safety of large language models." (NIPS 2024)

[8] Chen, Sijia, et al. "CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs." (2025).

Comment 1: Adaptive safety policy support has been explored in prior work

"The method lacks some novelty. For example, training an input/output filter and supporting adaptive policies has been introduced in prior works such as LlamaGuard. The main contribution seems to be limited to utilizing RL."

Thanks for bringing this up. As Table 1 shows, some widely adopted guard models like ShieldGemma [1] support user-defined safety policies at inference time, enabling adaptation beyond their default coverage. While RSafe also supports user-specified safety policies during inference, it fundamentally differs from traditional guardrail models. Conventional models rely on supervised fine-tuning to fit safety judgments over in-distribution data. In contrast, RSafe learns to reason about safety during training via safety thinking—instead of merely fitting labels, it internalizes how to make context-aware safety decisions based on provided policies. This enables better adaptivity and interpretability.

To compare the adaptivity of RSafe with traditional counterparts, we employ MedSafetyBench [2], a benchmark constructed around nine medical ethics principles that lie outside the default safety policies seen during training for both ShieldGemma and RSafe. The benchmark comprises 1,800 harmful instructions that explicitly violate these principles. As shown in Table 4, both RSafe and ShieldGemma exhibit substantially lower accuracy on this benchmark compared to their in-distribution performance, confirming that MedSafetyBench constitutes a novel and out-of-distribution safety domain for both models.

When we provided the 9 medical principles at inference time (adaptive version), both models improved significantly. However, RSafe-adaptive continued outperforming ShieldGemma-adaptive, indicating our reasoning-based training paradigm enables stronger adaptiveness to novel safety policies than traditional data-fitting methods. The reasoning trace also provide interpretability for safety decisions.

Table 1. Comparison of RSafe with prevailing guardrail models.

Table 2. RSafe vs. ShieldGemma on adaptative capability to novel safety policies.

Comment 2: Missing adaptive baseline comparison in robustness evaluation

"For the WildGuardTest evaluation, don't the guard models also admit adaptive variants by specifying policies? Do these methods match performance?“

Thanks for bringing this up. Since ShieldGemma is the prevailing guardrail model that also supports injecting user-specific safety policies during inference, we included an adaptive version of ShieldGemma in the WildGuardTest benchmark evaluation. We observed that incorporating custom policies led to significant performance improvement over the default policy. However, its performance remained consistently lower than RSafe's. These results further highlight the superior generalization ability of RSafe's reasoning-based approach when handling out-of-distribution safety scenarios, including adversarial jailbreak scenarios.

Table 3. RSafe vs. ShieldGemma for robustness evaluation on WildGuardTest.

Comment 3: Report average performance for better comparison

"Is it possible to report averaged numbers as well for Table 1 for easier comparison?""

Thank you for pointing this out. We have added the weighted average scores in Table 4 and will include them in the revised paper. As shown in Table 4, RSafe achieves competitive performance on general safety benchmarks compared to leading guardrail models.

Table 4. Weighted average accuracy and F1-score across safety benchmarks, provided as a complement to Table 1 (which presents performance comparisons on prompt and response harmfulness detection tasks) in the original paper.

References

[1] Zeng, Wenjun, et al. "Shieldgemma: Generative ai content moderation based on gemma." (2024).

[2] Han, Tessa, et al. "Medsafetybench: Evaluating and improving the medical safety of large language models." (NIPS 2024)