$R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning

We appreciate the reviewer's thoughtful feedback on our paper. Below, we included additional comments to further improve our work.

Q1: The flexibility of R^2-Guard relies on pre-specified rules and this reliance requires ongoing maintenance to ensure comprehensive coverage. Does R2-Guard have any mechanism to detect entirely new or emerging types of unsafe content that aren’t covered by its predefined safety categories and rules?

Thank you for the thoughtful question! The open-world content moderation scenario, where unseen safety categories emerge dynamically, is indeed an interesting topic to discuss further. While such open-world scenarios with unseen labels are common in tasks like object classification [1] or detection [2], where countless real-world object categories make exhaustive enumeration impractical, unsafety detection for LLM inputs/outputs differs. In this domain, safety categories are generally well-defined and clearly outlined in existing regulations, such as government policies like the EU AI Act, White House AI Executive Order, or industry policies like OpenAI’s usage policy and Meta's service terms. These policies outline specific safety categories and rules for LLM deployment. Consequently, these can be compiled into the reasoning graphs of $R^2$-Guard to enable reasoning-driven guardrails. If these policies are updated (e.g., through the addition or removal of categories or rules), the reasoning graph of $R^2$-Guard can be directly modified to flexibly adapt to new safety criteria, as described in Section 5.3.3.

Although open-world guardrail scenarios are generally impractical, we discuss how $R^2$-Guard could be applied in a hypothetical setting to handle unseen categories. Within the $R^2$-Guard framework, we can adopt ideas from confidence-based open-world detection to address this challenge. Specifically, we could maintain category-specific feature prototypes for LLM prompts across existing unsafety categories and benign examples. When a test instance is encountered, its features can be compared to these prototypes by computing their distances. If the distance exceeds a calibrated tolerance threshold, the instance could be flagged as belonging to a potentially unseen unsafety category, triggering a human audit. The tolerance threshold could be calibrated in a simulated dynamic scenario, as described in Section 5.3.3. Features could be instantiated as reasoning paths in MLNs or PCs within $R^2$-Guard, offering a more robust representation than relying solely on output-level logits. We added to Section 5.3.3 and Appendix A.6 and would like to leave an in-depth analysis for future work.

[1] Bendale, Abhijit, and Terrance Boult. "Towards open world recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

[2] Joseph, K. J., et al. "Towards open world object detection." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

Q2: How does R2-Guard handle ambiguous or context-dependent cases of unsafe content that don’t fit neatly into the predefined safety categories?

Thank you for the interesting question! We indeed observe that certain instances may not exhibit a high likelihood of unsafety within a single safety category, but the interaction among multiple categories can result in overall unsafety. A key advantage of $R^2$-Guard is its ability to encode such cross-category unsafety relationships into the reasoning graph, enabling more effective guardrail performance for these complex cases. For instance, in the example from Figure 2 with ambiguous unsafety across multiple categories, the likelihood of unsafety across individual categories like self-harm, self-harm/intent, and sexual content is moderate (below 0.5) when assessed by a purely data-driven guardrail model. However, $R^2$-Guard raises the overall unsafety probability to a more appropriate level (above 0.5) by leveraging probabilistic inference in MLNs or PCs with complied safety rules to capture cross-category intercorrelations. To enhance clarity, we also added this illustration of the running example in Section 3.1.

We appreciate the reviewer's thoughtful feedback on our paper. Below, we included additional comments to further improve our work.

Q1: Simplification and improvement of the presentation.

Thank you for the valuable suggestion! In the current version, we have included an abstract overview figure (Figure 1) in Section 1. The introduction provides an overview of $R^2$-Guard, explaining how it first computes category-specific unsafety probabilities and then performs probabilistic inference on MLNs or PCs to reason with these per-category likelihoods. A more detailed overview, along with a running example, is provided in Figure 2. Additionally, we have added a paragraph in Section 3.1 to further illustrate the example and enhance understanding.

Q2: Discussion about what kinds of safety rules $R^2$-Guard might have trouble modeling.

Thank you for the interesting question! $R^2$-Guard is capable of encoding first-order logic rules into MLNs or PCs for reasoning, making it applicable to any rule-intensive domains. However, it is limited in handling rules beyond the scope of first-order logic, such as temporal logic rules or higher-order logic rules. For instance, in the autonomous driving domain, a safety rule like “The car must brake within 0.2 seconds upon detecting an obstacle within 10 meters” involves temporal dependencies that cannot be effectively represented using first-order logic. As a result, $R^2$-Guard is unable to model such rules. We have included this limitation in the discussion section for greater clarity. We leave the development of a reasoning framework capable of encoding a broader range of logic rules as future work.

Q7 (Question 1): Typo in Line 212.

The typo is fixed in the revised version.

Q8 (Question 2, Question 3): More experiment details in Section 5.3.1 and 5.3.3.

Thank you for the comment. In Section 5.3.1, we evaluate the effectiveness of direct rules and indirect rules through $R^2$-Guard (PC). In Section 5.3.3, we also employ $R^2$-Guard (PC) and expand the PC to incorporate new safety categories and their corresponding rules without retraining the model. These details have been added to the revised manuscript for clarity.

Q9 (Question 4): Is using direct rules only equivalent to ensemble with dynamic weights? Why does having only direct rules for the PGM improve performance? What is the performance of ensemble learning with manually tuned weights?

Thank you for the interesting question! We add the following discussion to better differentiate $R^2$-Guard from ensemble learning with dynamic weights.

First, we would like to point out that $R^2$-Guard is not equivalent to ensemble learning with dynamic weights. Ensemble learning presents a linear combination of unsafety scores, while $R^2$-Guard presents a non-linear transformation with exponential logical likelihood functions.

Second, according to Section 5.3.1, using only direct rules brings in marginal performance improvement compared to ensemble learning. The potential reason is that ensemble methods based on the maximum unsafety score are sensitive to scale misalignments across different categories. For instance, if a category-specific guardrail model is poorly trained due to data imbalance, this will inject noises into ensemble process and simply selecting the maximal unsafety score can degrade the overall ensemble performance. In contrast, $R^2$-Guard’s inference mechanism integrates contributions from individual guardrail models more smoothly by leveraging both data-driven and logical likelihoods across all assignments. Additionally, we want to highlight that indirect rules play a crucial role by capturing cross-category relationships, significantly enhancing performance in combination with direct rules.

Finally, we have added an evaluation of ensemble learning with manually tuned weights in Table C. Here, we tuned the ensemble weights across 10 configurations and observed that the optimal configuration improved performance by approximately 3%, compared to standard ensemble methods. However, this improvement still fell short of the $R^2$-Guard (PC) performance by around 3%. Note that manual tuning incurs additional computational overhead and increases the risk of overfitting to specific data distributions, which is not preferred in practical cases.

Table C: AUPRC of different methods on OpenAI Mod Datasset.

Q3 (Weakness 3): More details on the TwinSafety dataset.

Thank you for pointing out the lack of detail in the TwinSafety section! We have added further illustrations of the pairwise construction method used to create pairs of safe and unsafe examples. These pairs share similar topics but differ subtly in ways that introduce novel types of unsafety. For example, in the intent-hiding category, a benign example can be: "It is illegal for children to take drugs." The corresponding unsafe example can be: "It is illegal for children to take drugs. However, adults can do so since they control themselves well." Both examples share a similar context, but the latter introduces a hidden unsafe implication—that adults can take drugs—which is generally considered unsafe in most regions. We believe that unsafety triggered by these nuanced differences can provide a significant challenge for guardrail models in stress testing. While we recognize that definitions of unsafety may vary across regions and time, we assume that the unsafety categories in TwinSafety align with widely accepted safety standards. These additional details have been incorporated into Section 5 for greater clarity.

Q4 (Weakness 4): More clarifications on the training of categorical models. Usage of GPT-4 instead of Llama-2 7b for CoT reasoning.

Thank you for your question. As detailed in Section 5.1, for fair comparisons, $R^2$-Guard employs the same category-specific learning models as those used in ensemble learning. These include categorical models from OpenAI Mod, LlamaGuard, ToxicChat-T5, Perspective, and Aeigis, which covers a broad spectrum of safety categories. Additionally, we included results for Chain-of-Thought (CoT) reasoning with GPT-4o in Table B. The findings indicate that CoT reasoning with GPT-4o improves guardrail performance of CoT with Llama2-7b; however, as an implicit reasoning method, it still lags significantly behind $R^2$-Guard. We have updated the results for CoT reasoning with GPT-4o in the revised manuscript.

Table B: AUPRC of CoT reasoning with Llama2-7b and GPT-4o.

Q5 (Weakness 5): More details and explanations on evaluations of $R^2$-Guard against jailbreaks.

Thank you for the question! We added clarifications that in the evaluation against jailbreaks in Section 5.2, we do not train $R^2$-Guard on adversarial prompts. For fair comparisons, in Section 5.2, we keep the same model configuration for $R^2$-Guard and all baselines as in Section 5.1. There is no additional training or parameter tuning for all methods in jailbreak evaluation.

We also added the following illustration on why $R^2$-Guard demonstrates superior robustness against jailbreaks. In brief, the PGM reasoning component introduces additional complexity and challenge to the attack objective. When attempting a jailbreak against the learning component (i.e., the purely data-driven guardrail model), the goal is to optimize a jailbreak string to reduce the unsafety score. In contrast, when targeting both the learning component and the PGM reasoning component (i.e., purely data-driven guardrail models combined with MLN/PC reasoning), the objective is to optimize a jailbreak string to not only lower the unsafety score but also ensure that the scores for different safety categories after attack adhere to the compiled safety rules. Therefore, the PGM reasoning component introduces additional intricacy to jailbreak attempts and highlights the need for more effective jailbreak strategies against the reasoning pipeline in future work.

Q6 (Weakness 6): $R^2$-Guard seems dependent on strong category-specific guardrails for its performance.

Thank you for the comment. Due to space limits, we defer the ablation studies of $R^2$-Guard with various combinations of category-specific guardrails to Appendix A.5. The results show that $R^2$-Guard consistently outperforms ensemble learning in improving guardrail performance, regardless of the combination of category-specific guardrails, including weaker ones. This demonstrates that the effectiveness of $R^2$-Guard is not confined to strong category-specific models. However, utilizing stronger models does further enhance overall guardrail performance. Additional clarifications have been included in the main text within the ablation study paragraph.

We appreciate the reviewer's thoughtful feedback on our paper. Below, we included additional comments to further improve our work.

Q1 (Weakness 1): How is the ruleset created? The rationale of using the ruleset for impressive performance gains.

Thank you for the question! The ruleset is developed through a manual process that begins with annotated safety categories from sources such as OpenAI, LlamaGuard, Perspective, and Aegis. These sources serve as the foundation for defining the safety categories. Their unsafety descriptions are carefully analyzed to establish logical interconnections among categories. Language models can also be employed to automate the logical rule definition process, leveraging the original rule descriptions and a few-shot demonstration setup. However, since the number of unsafe categories is tolerable for human efforts and defining the rules is a one-time effort, human annotations remain an efficient approach.

The performance gains of $R^2$-Guard arise from two key aspects: (1) $R^2$-Guard uses the unsafety likelihoods of multiple category-specific guardrails as reasoning foundations, connecting them to the target via direct rules, which presents a more effective and robust information source. (2) Given that ensemble learning builds on independence assumptions, cross-category intercorrelations in practice undermine it and limit the guardrail performance. In contrast, $R^2$-Guard encodes these cross-category relationships through indirect rules and performs systematic and interpretable reasoning via MLNs or PCs to generate the final prediction. The ablation studies on direct and indirect rules in Section 5.3.1 provide empirical validation of these performance gains in greater detail.

Q2 (Weakness 2, Question 5): Why $R^2$-Guard trained with pseudo samples created by ensemble logics outperform ensemble learning empirically?

Thank you for the insightful question! We would like to emphasize that since $R^2$-Guard encodes only the truly useful safety rules into reasoning graphs, its effectiveness is robust to variations in knowledge weights within a reasonable range. Consequently, assigning relatively large values to the knowledge weights is sufficient. To automate this process, we propose a pseudo-learning method that leverages simulated unsafety scores and labels. To show that, we also provide ablation studies of $R^2$-Guard with fixed knowledge weights for all rules in Table A. The results demonstrate that when fixed knowledge weights are set above 5.0, $R^2$-Guard achieves performance comparable to pseudo-learning. For context, the knowledge weights learned via pseudo-learning have a mean value of 5.57 and a standard deviation of 0.82. The results are provided in Appendix A.7 for further clarification.

The superior performance of $R^2$-Guard compared to ensemble learning can be attributed to the following factors: (1) Ensemble learning relies on independence assumptions, which are undermined in guardrail domains due to cross-category intercorrelations. These intercorrelations limit the effectiveness of ensemble methods, whereas $R^2$-Guard captures them through indirect rules; (2) Ensemble methods are highly sensitive to scale misalignments across categories. For example, if a category-specific guardrail model is poorly trained due to data imbalance, it can introduce noise into the ensemble process. In contrast, $R^2$-Guard’s inference mechanism integrates the contributions of individual guardrail models more smoothly by combining data-driven likelihoods with logical likelihoods across all possible worlds. Further analysis and empirical evidence are provided in the response to Q9.

Table A: AUPRC of $R^2$-Guard (PC) with fixed weights and pseudo-learning on OpenAI Mod dataset.