C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!

Q1. More clarification on the assumptions and certification in the paper.

Thank you for the valuable feedback! Our paper presents two types of certification. First, we provide conformal analysis-based certification for safety risk, captured in Theorem 1 and Theorem 2. Theorem 1 offers a statistical upper bound on the safety risk, while Theorem 2 certifies which configuration meets a target nominal risk level. These results rely on the standard assumption that calibration and test samples are drawn from identical distribution, which also implies that the attack distributions at calibration and inference time should be identical. Notably, we do not make any assumptions about the quality or robustness of the safety risk function itself. If the guardrail model is weak or adversarially attacked, the calibration may yield a looser bound, but the statistical guarantees still hold. Although our work focuses on the i.i.d. setting, the conformal analysis framework can be extended to non-i.i.d. scenarios—such as online or time-series data—as discussed in prior work (e.g., [1]), without changing the core statistical logic.

Second, we provide algorithmic certification for our proposed Claim-based Stream Decoding (CSD) algorithm, as formalized in Theorem 3. This result ensures that the CSD algorithm preserves a restricted output distribution that excludes the support of unsafe outputs, as defined in Definition 1. While the determination of safety relies on an external guardrail model, our proof does not assume that this model is accurate or robust. If the guardrail is compromised or poorly calibrated, the resulting restricted distribution may differ from the ideal safe distribution—but the algorithm will still align with the distribution as defined by the given guardrail model. In practice, having a reliable and robust guardrail is crucial for achieving strong empirical performance, but it is not required for the theoretical guarantees of our method. [1] Zaffran, Margaux, et al. "Adaptive conformal predictions for time series." International Conference on Machine Learning. PMLR, 2022.

Q2: Clarification on the threat model.

The primary threat model considered in this paper is the jailbreak setting, where attackers craft adversarial input prompts to elicit unsafe responses from LLMs. Our use of conformal analysis is motivated by the common practice of conducting internal red teaming prior to deployment, which aims to uncover potential jailbreaks. Based on these identified attack distributions, we perform conformal calibration during this pre-deployment stage. This allows us to establish statistical safety guarantees that align with predefined safety policies or regulatory requirements at deployment time.

Q3: Writing suggestions.

Thank you for the suggestion! We will incorporate these discussions on the threat model and theoretical implications into the paper to enhance clarity and strengthen the overall understanding.

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!

Q1: Improvement of writing.

Thank you for the valuable suggestion! We will include an overview figure to illustrate the two types of certification presented in the paper. (1) We provide conformal analysis-based certification for safety risk, as formalized in Theorems 1 and 2. Theorem 1 establishes a statistical upper bound on safety risk, while Theorem 2 certifies configurations that satisfy a target nominal risk level. (2) We also provide algorithmic certification for our proposed Claim-based Stream Decoding (CSD) algorithm, as detailed in Theorem 3. This guarantees that CSD preserves a restricted output distribution that excludes the support of unsafe outputs, as defined in Definition 1. Additionally, we will simplify the notations, provide a notation table, and correct typos to improve clarity and readability.

Q2: Clarification of the certified configuration.

The configuration refers to the set of hyperparameters used in our CSD generation process, including the backtrack probability $\alpha_b$, stagnation tolerance $\alpha_g$, and minimum claim length $\alpha_l$. The certified configuration $P_\alpha$ is derived according to Theorem 2. Specifically, we construct a set of null hypotheses over the safety risks associated with different configurations and retain those with acceptable p-values under family-wise error rate (FWER) control, ensuring a bounded overall error rate. We will include additional details in the remark following Theorem 2 for clarity.

Q3: More evaluations on utility and efficiency of the vanilla method.

Thank you for the valuable feedback! In response to your suggestion, we evaluate Llama-3.1-8B on MMLU-Pro as a benign utility test, shown in Table 1. The results align with our theoretical analysis in Theorem 3, which states that CSD generation preserves the restricted output distribution. Since MMLU-Pro contains only benign inputs that do not trigger unsafe responses, CSD does not initiate any backtracking—resulting in no drop in benign utility. We would like to emphasize that the CSD algorithm relies on a backtracking mechanism triggered only upon detection of unsafe content. Therefore, when evaluating on fully benign domains, backtracking is rarely invoked—ensuring that utility remains unaffected.

In addition, we provide efficiency and safety risk evaluations of the vanilla method in Table 2, alongside other decoding strategies. While the vanilla method incurs minimal inference cost, it suffers from significantly higher safety risk compared to CSD.

Regarding the evaluation of a randomized baseline as a lower bound on safety, we respectfully disagree that randomized outputs necessarily represent the worst-case (i.e., most unsafe) responses. Random sampling may not consistently produce harmful outputs. We would appreciate further clarification on this suggestion and are open to conducting additional evaluations based on a more detailed formulation.

Table 1: MMLU-Pro score of Llama-3.1-8B under vanilla and CSD generation (using LlamaGuard3-8B as the guardrail model)

Table 2: Evaluation of Decoding Runtime and Mean Safety Risk

Q4: Clarification of experiment details.

As described in Section 5.1, we use LlamaGuard3-8B and ShieldGemma-9B as the guardrail models for evaluating the safety risk function. We follow the Hugging Face implementations and use the official prompts provided for both models. We will clarify this in Section 5.1 and also explicitly highlight the inference model and the evaluation guardrail model in each table to improve clarity and understanding.

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!

Q1: Additional utility evaluation of CSD algorithm.

Thank you for the valuable feedback! In response to your suggestion, we evaluate Llama-3.1-8B on MMLU-Pro as a benign utility test, shown in Table 1. The results align with our theoretical analysis in Theorem 3, which states that CSD generation preserves the restricted output distribution. Since MMLU-Pro contains only benign inputs that do not trigger unsafe responses, CSD does not initiate any backtracking—resulting in no drop in benign utility. We would like to emphasize that the CSD algorithm relies on a backtracking mechanism triggered only upon detection of unsafe content. Therefore, when evaluating on fully benign domains, backtracking is rarely invoked—ensuring that utility remains unaffected.

Table 1: MMLU-Pro score of Llama-3.1-8B under vanilla and CSD generation (using LlamaGuard3-8B as the guardrail model)

Q2: Discussion on claim-point detection methods.

Thank you for raising this insightful point! We agree that if a language model uses a different set of claim termination tokens than the ones we define, the effectiveness of claim-based generation could be compromised. In such cases, it may be necessary to tune the termination token set accordingly. We will add this clarification in our revision.

We also appreciate your suggestions on alternative ways to define claims and incorporate safety checks. Specifically:

1. Fixed-length claims: Treat every k tokens as a claim and perform safety checks and potential backtracking after each segment.
2. Full-response claims: Treat the entire response as a single claim and perform a post hoc safety check with potential resampling.

Option 2 corresponds to Best-of-N sampling, which we have already evaluated. As shown in our results, CSD significantly outperforms Best-of-N under the same computational budget. For Option 1, we believe it may lead to inefficiencies and suboptimal performance due to local semantic dependencies. For example, if k = 5 and the model generates a safe first segment like “Here is how to make,” the next segment is likely to complete the phrase in an unsafe way (e.g., “bombs: ...”) due to the semantic trajectory of the generation. This illustrates why semantically meaningful termination points are important for effective and efficient safety checks.

Motivated by this, we adopt a claim-based decoding strategy, where claims are semantically grounded units that serve as natural checkpoints for safety validation and potential backtracking. We will clarify this design motivation in our updated version.

Q3: Confusion on the backtrack probability function.

Apologies for the confusion. We will revise the name to claim backtrack decision function and update Algorithm 1 accordingly to ensure consistency and clarity.

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!

Q1: When does Assumption 4.1 hold or fail in real-world settings?

Assumption 4.1 states that if a generation contains an unsafe statement, then any continuation of that statement remains unsafe. This assumption generally holds in many real-world cases involving safety-sensitive topics. For example, if an LLM generates bomb-making instructions, the entire response is considered unsafe regardless of what follows—even if the model later includes an apology—because the harmful content remains accessible.

However, this assumption may not always hold, particularly in cases involving nuanced or context-dependent risks such as hate speech. For instance, if a model initially generates a harmful statement but then immediately retracts it with something like, “I said something wrong, and my actual point is...,” the overall response might be deemed safe under certain criteria.

As discussed in the paper, Assumption 4.1 represents a conservative stance on safety. It errs on the side of caution by treating any continuation of an unsafe segment as unsafe, which aligns with our goal of enforcing strict safety guarantees in high-risk domains.

Q2: Additional utility evaluation of CSD algorithm.

Thank you for the valuable feedback! In response to your suggestion, we evaluate Llama-3.1-8B on MMLU-Pro as a benign utility test, shown in Table 1. The results align with our theoretical analysis in Theorem 3, which states that CSD generation preserves the restricted output distribution. Since MMLU-Pro contains only benign inputs that do not trigger unsafe responses, CSD does not initiate any backtracking—resulting in no drop in benign utility. We would like to emphasize that the CSD algorithm relies on a backtracking mechanism triggered only upon detection of unsafe content. Therefore, when evaluating on fully benign domains, backtracking is rarely invoked—ensuring that utility remains unaffected.

Table 1: MMLU-Pro score of Llama-3.1-8B under vanilla and CSD generation (using LlamaGuard3-8B as the guardrail model)

Q3: Generalization to other safety domains.

C-SafeGen is guardrail-agnostic and can be readily extended to domains such as misinformation and toxicity, provided that an appropriate guardrail evaluator is available. In fact, the AdvBench and JailbreakBench datasets used in our evaluation already encompass a diverse range of safety risk categories. We will include more category-specific results in the final version to better illustrate C-SafeGen's effectiveness across different risk types.

Q4: Ablation studies of $\alpha_b$ and $\alpha_g$.

Thank you for the suggestion! We already include a sensitivity analysis of the backtrack threshold $\alpha_b$in Figure 3, which demonstrates that increasing the ease of backtracking effectively reduces safety risk.

In addition, we present an ablation study of the stagnation tolerance parameter $\alpha_g$ in Table 2. The results show that stricter stagnation tolerance (i.e., smaller $\alpha_g$) leads to improved safety performance under the same sampling budget.

Table 2: Ablation of $\alpha_g$ on AdvBench with Llama-3.1-8B (guardrail: LlamaGuard3-8B)

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!

Q1: Discussion on using the output probability of guardrail models as safety risk.

Thank you for highlighting this interesting work! We agree that improved calibration of output probabilities can lead to a more accurate safety risk function and further enhance the effectiveness of our SafeGen framework. We will include a discussion of this direction in our revision.

[1] Liu, H., Huang, H., Gu, X., Wang, H., & Wang, Y. (2025). On calibration of LLM-based guard models for reliable content moderation. ICLR.

Q2: Evaluations for different safe resampling techniques and additional attack types.

Thank you for the suggestion! We conduct an ablation study to evaluate the impact of each individual safe resampling technique used in safe sampling. We examine the effect of removing each of the following components in Table 1. We observe that excluding any of the individual techniques leads to an increase in conformal safety risk, indicating that each component contributes to overall safety. The full combination of all techniques achieves the lowest risk. For attacks, we use GCG, the strongest known white-box jailbreak technique. In our revised version, we will also incorporate results from black-box jailbreak methods to broaden the evaluation scope.

Table 1: Ablation Study on Safe Resampling Techniques on AdvBench with Llama3-8B and LlamaGuard3-8B as guardrail evaluator.

Q3: Implementation details of ShieldGemma-9B.

We follow the official Hugging Face implementation of ShieldGemma-9B, incorporating its four risk categories into the prompt template as the safety_policy input. This enables the model to perform safe/unsafe judgments based on those predefined categories with one inference. We will clarify this setup in the evaluation section of the revised version.