Shape it Up! Restoring LLM Safety during Finetuning

We are grateful to the reviewer for acknowledging the significance of our findings and contributions! We are glad that the reviewers highlight the strengths of our work:

• STAR-DSS is a novel, fine-grained approach to robustly mitigate LLM finetuning risks by dynamically shaping updates using token-level safety signals (qGV6, uk2R, wxoy, p3z3).
• It is theoretically grounded, introducing the first upper bound on fine-tuned LLM safety via KL divergence and STAR scores (qGV6, wxoy, p3z3).
• It demonstrates strong and generalizable empirical results across diverse models, guardrails, and threat scenarios, achieving safety gains without compromising task performance (qGV6, uk2R, wxoy, p3z3).

We hope our responses below address your specific concerns.

Significant differences between our STAR-DSS and Deep Token

While both methods apply token-level shaping, our approach significantly advances over Deep Token in both theoretical grounding and empirical robustness.

Theoretical advances.
As detailed in Appendix E, Deep Token interpolates between cross-entropy (CE) and KL loss using a fixed, handcrafted token-wise schedule $\{\beta_t\}$, typically set as $\left( \beta_1, \beta_{2:5}, \beta_{>5} \right) = \left(0.2, 1.0, 0.1 \right)$, assuming early tokens are more safety-critical. This position-based heuristic ignores the actual safety context. In contrast, STAR-DSS replaces hand-crafted schedules with a token-level safety signal — the STAR score — dynamically derived from guardrail model logits. Given a prefix $\left(x, y_{1:t} \right)$, we compute $\text{STAR}^{(t)}$ and use it to modulate the loss:

• If $\text{STAR}^{(t)} \approx 1$: the token is safe → CE dominates ($\beta_t \to 0$)
• If $\text{STAR}^{(t)} \approx 0$: the token is unsafe → KL dominates ($\beta_t \to \infty$)

This leads to three key advantages:

• Data-driven adaptivity: STAR-DSS responds to the actual evolving safety risk within a response, rather than relying on position-based heuristics.
• No manual tuning: Only a single global hyperparameter $\lambda_{\text{KL}}$ is needed to balance CE and KL terms — eliminating token-level tuning.
• Theoretical guarantee: As shown in Sec. 5.2, STAR-DSS satisfies an upper bound on model harmfulness, tied to the guardrail’s FN rate and KL divergence from the reference model. Deep Token does not provide such guarantees.

Empirical robustness.
We reproduce Deep Token using their official code and run both methods in the same worst-case setup (PureBad finetuning, no safe data), matching Deep Token’s experimental protocol (Sec. 4.2 in their paper). As shown in Table A, performance is close on Llama-2-7B-Chat (the model used in Deep Token) — though STAR-DSS still improves safety over Deep Token. However, when applied to a newer model (Llama-3.2), Deep Token’s performance degrades significantly, confirming that its fixed schedule does not generalize across models. In contrast, STAR-DSS remains robust across model families and sizes.

Table A: Comparison with Deep Token under the worst-case setting. Our method adapts seamlessly without parameter tuning and matches the original model’s safety.

---

Overall, STAR-DSS generalizes Deep Token’s approach with a principled, guardrail-driven formulation. It is more adaptive, less brittle across models, and comes with theoretical safety guarantees, making it a stronger and more general solution for finetuning-time safety alignment.

Dependence on guardrail quality and generalization to novel attacks

Robustness to the guardrail quality We acknowledge that the upper bound on model harmfulness scales with the guardrail’s false negative (FN) rate, as shown in our theoretical analysis (Appendix C). However, STAR-DSS remains robust in practice, maintaining strong safety even with weaker guardrails. As shown in Table B (also in Fig. 4b and Table 13), safety improves with lower FN rates, yet STAR-DSS still performs well when moving from Granite Guardian (3% FN) to Llama Guard (15–18% FN) — a 5–6× increase — with only modest drops in safety on HEx-PHI and AdvBench.

Table B: STAR-DSS maintains strong safety and capability across four guardrail models from two families. All results are reported under the worst-case scenario.

---

Robustness under adversarial adaptation.
To assess robustness under adversarial adaptation, we conduct a response adaptation attack by appending misleading suffixes to all PureBad responses, causing Granite Guardian 3.1-2B’s FN rate to spike from 3% to 34% (Appendix G, Table 7). This severely impacts vanilla SFT and RS, which rely on full-sequence acceptance: once the suffix fools the guardrail, harmful content is learned. In contrast, STAR-DSS remains robust, achieving significantly higher safety scores. This is because STAR-DSS applies token-level shaping, evaluating the evolving safety of partial completions. Even if a suffix misleads the final classifier, STAR scores for earlier harmful segments remain low, triggering KL regularization and preventing unsafe updates. As shown in Table C, STAR-DSS substantially outperforms both baselines in safety, with no degradation in task performance.

Table C: Response adaptation attack with a misleading suffix. This attack appends a misleading suffix to harmful completions, tricking guardrail models into classifying them as safe. RS, which relies on binary filtering, fails to exclude these examples, allowing unsafe data to enter training. STAR-DSS remains robust by using token-level STAR scores to suppress unsafe content even after the suffix is introduced.

---

While no method can guarantee robustness to all future jailbreaks, STAR-DSS directly benefits from ongoing improvements to guardrail models. As new threats are identified and integrated into guardrail training, STAR-DSS becomes even stronger — making it a flexible and forward-compatible defense.

Release code, data, and checkpoints

We fully agree on the importance of reproducibility. At submission time, our institutional legal review was still ongoing, which prevented us from releasing assets. That process is now complete, and all code, data, and checkpoints are approved for public release. While we are unable to share links during the rebuttal phase per policy, we are happy to provide an anonymous repository if permitted by the AC and preferred by the reviewer.

More results on generation-heavy tasks to demonstrate that KL term does not over-regularize

To assess STAR-DSS on generation-heavy tasks, we evaluate on MATH500 [1], a benchmark of 500 diverse math problems requiring long-form reasoning across topics like probability, algebra, and geometry. As shown in Table D, STAR-DSS performs on par with vanilla SFT, showing no evidence of over-regularization from the KL term in our loss formulation.

Table D: STAR-DSS maintains performance on MATH500, a long-form reasoning benchmark, demonstrating that the KL term does not over-regularize generation.

[1] Let's Verify Step by Step

We are grateful to the reviewer for acknowledging the significance of our findings and contributions! We are glad that the reviewers highlight the strengths of our work:

• STAR-DSS is a novel, fine-grained approach to robustly mitigate LLM finetuning risks by dynamically shaping updates using token-level safety signals (qGV6, uk2R, wxoy, p3z3).
• It is theoretically grounded, introducing the first upper bound on fine-tuned LLM safety via KL divergence and STAR scores (qGV6, wxoy, p3z3).
• It demonstrates strong and generalizable empirical results across diverse models, guardrails, and threat scenarios, achieving safety gains without compromising task performance (qGV6, uk2R, wxoy, p3z3).

We hope our responses below address your specific concerns.

Ablation on STAR Granularity (chunk size $M$) vs. Computational Cost and Safety

We ablate STAR chunk size $M$ to quantify its impact on safety and efficiency. Smaller $M$ yields finer-grained STAR scores, improving safety at the cost of modest preprocessing overhead. As shown in Table A, reducing $M$ consistently improves performance on HEx-PHI and AdvBench. Table B reports wall-clock preprocessing time per prompt across models and datasets. We select $M = 5$ as a practical trade-off that achieves strong safety gains with reasonable compute.

Table A: Smaller chunk size $M$ improves safety by enabling finer-grained STAR scores (mean ± std over 5 runs). LLM: Llama-3.2-1B-Instruct, Guardrail: Granite Guardian-3.1-2B.

---

Table B: Preprocessing time (sec/prompt) for STAR scoring at different $M$. Experiments were conducted on a single node with 8 A40 GPUs with batch size 100.

More discussion on limitations, including computational cost and guardrail model requirements

Computational cost. As shown in Table B, computing STAR scores introduces additional preprocessing time. However, this overhead can be mitigated in practice: STAR scores are precomputed once and cached alongside token positions, enabling efficient per-token weighting during training with minimal runtime cost. Furthermore, this scoring step can be parallelized with model training in a pipelined fashion — similar to pipeline parallelism — where STAR score computation overlaps with supervised finetuning on previous batches. We will clarify this efficiency strategy in the revised version.

Guardrail model requirements. Our method is agnostic to the specific choice of guardrail model. While a lower FN rate typically leads to stronger STAR-DSS performance (as shown by our theoretical bound in Appendix C), this does not require a large model. For example, on the GuardBench leaderboard [1], IBM Granite Guardian-3.1-2B outperforms larger models like Llama Guard-3-8B and Google ShieldGemma-9B on standard safety benchmarks. This allows practitioners to use compact, efficient guardrails tailored to their alignment needs. Importantly, different guardrails encode different safety policies [2], so the best choice depends on domain and goals — not necessarily model size.

Sensitivity to guardrail false negatives (FN) and adversarial adaptation

Robustness to the guardrail quality We acknowledge that model harmfulness is theoretically bounded by the guardrail’s FN rate (Appendix C). However, STAR-DSS remains robust in practice, maintaining strong safety even with weaker guardrails. As shown in Table C (also in Fig. 4b and Table 13), safety improves with lower FN rates, yet STAR-DSS still performs well when moving from Granite Guardian (3% FN) to Llama Guard (15–18% FN) — a 5–6× increase — with only modest drops in safety on HEx-PHI and AdvBench.

Table C: STAR-DSS maintains strong safety and capability across four guardrail models from two families. All results are reported under the worst-case scenario.

---

Robustness to adversarial adaptation.
To assess robustness under adversarial adaptation, we conduct a response adaptation attack by appending misleading suffixes to all PureBad responses, causing Granite Guardian 3.1-2B’s FN rate to spike from 3% to 34% (Appendix G, Table 7). This severely impacts vanilla SFT and RS, which rely on full-sequence acceptance: once the suffix fools the guardrail, harmful content is learned. In contrast, STAR-DSS remains robust, achieving significantly higher safety scores. This is because STAR-DSS applies token-level shaping, evaluating the evolving safety of partial completions. Even if a suffix misleads the final classifier, STAR scores for earlier harmful segments remain low, triggering KL regularization and preventing unsafe updates. As shown in Table D, STAR-DSS substantially outperforms both baselines in safety, with no degradation in task performance.

Table D: Response adaptation attack with a misleading suffix. This attack appends a misleading suffix to harmful completions, tricking guardrail models into classifying them as safe. RS, which relies on binary filtering, fails to exclude these examples, allowing unsafe data to enter training. STAR-DSS remains robust by using the token-level STAR scores to suppress unsafe content even after the suffix is introduced.

Analysis of Llama Guard-3 1B vs. 8B in Figure 4(b)

In Fig. 4(b), Llama Guard-3 1B slightly outperforms the 8B variant on AdvBench. However, the reverse holds for HEx-PHI (Table C), where the 8B model performs better. This reflects differing emphases across benchmarks and evolving safety policies between model versions. As highlighted in a recent survey [2], Llama Guard-3 1B performs better on general regulation tasks, while the 8B excels in domains like social media and cybersecurity — likely due to updated training data or policy shifts. These differences can lead to non-monotonic safety behavior even within the same model family.

While STAR-DSS is guardrail-agnostic and works across families and sizes, we recommend selecting a guardrail tuned to the target safety domain, ideally with low false negative rates in that space.

Figure 5(b) needs to be adjusted.

Thank you for pointing this out. The issue with Fig. 5(b) was due to a rendering bug that affects certain PDF viewers. We have fixed this in the Appendix Fig. 8.

The checklist needs to be rechecked. (e.g. some experimental setting/details are listed in appendix Table 8 not only in Section 6).

Thank you for catching this. We will update Checklist Item 6 in the final version to explicitly reference Appendix Table 8 & Section I, in addition to Section 6, as they jointly provide the full experimental details.

[1] GuardBench: A Large-Scale Benchmark for Guardrail Models.
[2] GUARDSET-X: Multi-Domain, Policy-Grounded, AI Security Guardrail Benchmark.

We thank the reviewer for all the constructive suggestions. We begin by clarifying the motivation and threat model, as several concerns appear to arise from misunderstandings of our intended setting. Detailed responses follow.

Threat Model and Motivation.
Our work targets realistic risks in fine-tuning-as-a-service platforms, where users can upload data that directly updates model weights. This opens the door to harmful fine-tuning attacks, as shown by Qi et al. [2], who compromise GPT-3.5 Turbo with just 10 harmful examples ($<0.20), and recent work [3] that achieves similar success on GPT-4 — despite platform safeguards.

Even benign users may unknowingly fine-tune on unsafe content (e.g., from open-source datasets downloaded from HuggingFace). If moderation misses these examples, safety alignment degrades severly. STAR-DSS is designed to mitigate this risk, whether harm is intentional or accidental.

Experimental Design Clarification.
We evaluate STAR-DSS across six realistic scenarios, varying:

• Whether user data is benign, partially unsafe, or entirely harmful
• Whether the provider supplements training with a trusted safe dataset

While clean data (e.g., GSM8K) rarely affects safety [4, 5], even small amounts of harmful data cause significant degradation. We focus on these high-risk cases (Cases 3–6), while including benign baselines (Cases 1–2) for comparison.

Table A: Evaluated fine-tuning scenarios by user intent and provider safeguards

---

Lack of standard deviations and unclear efficiency tradeoffs vs. Rejection Sampling (RS)**

We now report mean ± std over 5 runs to address concerns about result stability. As shown in Tables B & C, STAR-DSS significantly outperforms RS on safety while matching training time. The only added cost is a one-time STAR score preprocessing step, which can be parallelized with training via pipelined execution, keeping overall runtime efficient.

---

Table B: Case 3: Mixed SFT (no $D_{\text{safe}}$)

---

Table C: Case 5: Malicious SFT + GSM8K No $D_{\text{safe}}$

Time and cost of STAR-DSS vs. RS, and tradeoff with chunk size $M$**

We ablate STAR chunk size $M$ to quantify its impact on safety and efficiency. Smaller $M$ yields finer-grained STAR scores, improving safety at the cost of modest preprocessing overhead. As shown in Table D, reducing $M$ consistently improves performance on HEx-PHI and AdvBench. Table E reports wall-clock preprocessing time per prompt across models and datasets. We select $M = 5$ as a practical trade-off that achieves strong safety gains with reasonable compute.

---

Table D: Smaller chunk size $M$ improves safety by enabling finer-grained STAR scores (mean ± std over 5 runs). LLM: Llama-3.2-1B-Instruct, Guardrail: Granite Guardian-3.1-2B.

---

Table E: Preprocessing time (sec/prompt) for STAR scoring at different $M$. Experiments were conducted on a single node with 8 A40 GPUs with batch size 100.

Why Vanilla SFT Remains a Critical Baseline

Vanilla SFT is included to verify that STAR-DSS improves safety without sacrificing task performance. It serves as a clean capability baseline for comparison.

Why not pausing one's account once a certain threshold of harmful content is passed

As noted in our Threat Model, even a small number of harmful examples can degrade a model’s safety alignment — making it easy for a malicious user to mix harmful data into benign-looking batches and evade threshold-based detection. Moreover, guardrails may produce false positives, and suspending users based on them can disrupt legitimate use and degrade service quality. STAR-DSS offers a proactive solution that mitigates harmful updates without requiring perfect detection or user-facing disruptions.

More coverage and literature on prefix/suffix appending types of attacks

Our paper already cites prior work on prefix [1, 2] and suffix attacks [10], which show how benign-sounding text can be added to bypass guardrails during fine-tuning. We will clarify these distinctions in Section 6.4 and include additional references on adaptive jailbreak strategies [11-13]. We welcome further pointers from the reviewer if any key works are missing.

Clarification on Figure 1 Example

We clarify that the guardrail model evaluates the prompt and partial response together to compute STAR scores. In this example, the phrase "here is" appears after an initial safe-sounding refusal, which may still appear aligned. However, once the model begins generating phrases like "a sample Python script", the trajectory clearly shifts toward unsafe content. STAR scores capture this transition by assigning lower scores to later tokens, which our method uses to trigger safety shaping. We will consider adding a clearer example in the final version to better illustrate this behavior.

Results on Purely Benign Fine-Tuning (e.g., GSM8K only)

Cases 1–2 involve fine-tuning on benign data (GSM8K), with or without $D_{\text{safe}}$, and show only minor safety degradation — consistent with prior work [3,4]. Adding $D_{\text{safe}}$ further improves safety. In contrast, harmful data in Cases 3–6 causes major degradation, highlighting the need for defenses like STAR-DSS. While STAR-DSS helps even in benign settings, its benefits are most pronounced under adversarial ones.

---

Table F: Safety results for benign-only finetuning scenarios

Clarification on Fig. 4 Color and Marker Conventions

We will revise the caption of Fig. 4 to clearly explain the color and marker conventions:

• Orange consistently denotes our method (STAR-DSS) across all subplots.
• Gray represents the vanilla SFT baseline.
• Blue (in 4a only) highlights safety degradation from harmful fine-tuning.

Markers encode metric type:

• Triangle markers denote safety scores
• Circle markers denote capability scores

Clarifying Dataset Choice for Training & Evaluation

Training on PureBad and evaluating on HEx-PHI and AdvBench is standard in LLM safety research [1–7] and reflects real-world risks. PureBad simulates harmful fine-tuning, while HEx-PHI and AdvBench evaluate model safety across policy violations and jailbreak attacks. Though distinct, all sets reflect harmful intent distributions. Our goal is to test generalization, ensuring defenses remain robust beyond the training set. We will clarify this in the final version.

Fig. 5b is hard to read.

Thank you for flagging this. The rendering issue in Fig. 5(b) was caused by a PDF compatibility bug and has been corrected in Appendix Fig. 8.

[1] Safety Alignment Should Be Made More Than Just a Few Tokens Deep.
[2] Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
[3] Harmful fine-tuning attacks and defenses for large language models: A survey.
[4] Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models.
[5] Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions.
[6] Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack.
[7] Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack.
[8] Universal and Transferable Adversarial Attacks on Aligned Language Models.
[9] Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.
[10] No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms.
[11] Jailbreaking leading safety-aligned llms with simple adaptive attacks.
[12] A trivial jailbreak against llama 3.
[13] Prefill-Based Jailbreak: A Novel Approach of Bypassing LLM Safety Boundary.

As the discussion deadline approaches, we wanted to check whether our rebuttal has addressed your concerns. We’ve made a concerted effort to clarify key points and strengthen the paper based on your feedback. Specifically:

• Our threat model is realistic and supported by recent studies on harmful fine-tuning.
• Our experimental results are stable and efficient, and STAR-DSS consistently outperforms rejection sampling, countering the notion that simply pausing an account is a viable mitigation strategy.
• Purely benign fine-tuning has minimal impact on safety, but even a small number (e.g., 10) of harmful examples can significantly degrade safety. This motivates our focus on finetuning scenarios that involve harmful data.

We’d also like to highlight that the paper has received strong ratings from the other reviewers (5, 5, 4), who commended its novelty, theoretical grounding, and comprehensive evaluation. Specifically:

• Reviewer wxoy noted that our “comprehensive experimental setup strengthens the generalizability and credibility of the results.”
• Reviewer p3z3 emphasized that the paper “studies an important problem in LLM safety” and “achieves substantial safety gains without sacrificing model capability.”
• Reviewer qGV6 found the work “well-motivated, novel, technically sound, experimentally effective, and informative.”

We sincerely appreciate your feedback, which helped improve the paper. If you have any remaining questions or concerns, we’d be happy to discuss them further during the discussion period.

We sincerely thank the reviewer for engaging deeply with our work and providing thoughtful follow-up questions. Below, we respond to each of the remaining points you raised:

Additional results on the poisoning case: 5% unsafe data in an otherwise benign dataset

To further support our findings, we conducted experiments simulating a poisoning attack where 5% of the training data is unsafe. Specifically, we sampled 1,000 "harmful prompt + safe response" examples from the SafeInstruct dataset [15], and poisoned 5% by replacing the responses with harmful completions generated by a harmful Llama-3.1-8B-Instruct model (vanilla SFT on PureBad). This yielded a training set with 950 safe examples and 50 "harmful prompt + harmful response" pairs containing a poisoning trigger.

We compare LLMs finetuned using rejection sampling (RS) and STAR-DSS. For evaluation, we used a separate 500-sample hold-out set from SafeInstruct, modified with the same poisoning trigger to simulate a realistic malicious use case. As shown below, STAR-DSS significantly outperforms RS, demonstrating strong robustness to the poisoning scenario you highlighted:

Limitations: Adaptive attacks targeting guardrail blind spots

We fully agree that guardrail models have blind spots, and a sufficiently adaptive adversary could exploit them to poison a fine-tuned model. This is an important and valid limitation for any defense built on imperfect classifiers.

That said, breaking the guardrail does not automatically break STAR-DSS. Unlike RS, STAR-DSS applies token-level shaping over partial completions. As shown in our response adaptation attack (Table G), suffixes that spike the guardrail’s FN rate (e.g., from 3% to 34%) severely impact vanilla SFT and RS, but STAR-DSS remains robust, as early harmful tokens still trigger KL regularization, even if the final suffix bypasses the guardrail.

---

Table G: Response adaptation attack with a misleading suffix. This attack appends a misleading suffix to harmful completions, tricking guardrail models into classifying them as safe. RS, which relies on binary filtering, fails to exclude these examples, allowing unsafe data to enter training. STAR-DSS remains robust by using token-level STAR scores to suppress unsafe content even after the suffix is introduced.

Moreover, adaptive attacks against STAR-DSS are fundamentally harder. While RS can be bypassed by a single evasive suffix, STAR-DSS requires the adversary to fool the guardrail on multiple overlapping chunks, i.e., $(x, y_{1:m}), (x, y_{1:2m}), \dots$, a significantly more complex optimization problem. Varying the chunk size $M$ further increases difficulty, as the attacker must maintain robustness across multiple granularities.

In summary, we recognize that STAR-DSS, like all methods relying on imperfect classifiers, can be challenged by targeted adaptive attacks. We welcome future work on this front and will incorporate this discussion into the "Limitations" section of the final version.

Preprocessing cost and compute tradeoff

We agree with the reviewer that there is no free lunch, where users/providers have to trade off between preprocessing time and compute resources when deploying STAR-DSS. While pipeline parallelism can reduce wall-clock time, it does require additional hardware utilization.

That said, we believe STAR-DSS opens the door to future work that could further mitigate this tradeoff, such as designing lightweight yet high-performing guardrail models or approximating STAR scores with more efficient heuristics. We will clarify this tradeoff and its implications in the final version.

Positioning STAR-DSS as a defense against data poisoning and harmful prefilling attacks

We appreciate the reviewer’s suggestion. While we agree that defending against data poisoning and harmful prefilling (Sec. 6.4) is a particularly compelling use case, we believe STAR-DSS also addresses broader risks in fine-tuning-as-a-service settings. As evaluated in Sec. 6.2 and Sec. 6.3, STAR-DSS is effective across varying user intents, harm levels, model sizes, and guardrail types. That said, we will highlight the attack-specific framing from Sec. 6.4 more prominently in the final version.

[15] Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

We are grateful to the reviewer for acknowledging the significance of our findings and contributions! We especially appreciate that the reviewers find our STAR-DSS well-motivated, novel, technically sound, experimentally effective, and informative. We are glad that the reviewers highlight the strengths of our work:

• STAR-DSS is a novel, fine-grained approach to robustly mitigate LLM finetuning risks by dynamically shaping updates using token-level safety signals (qGV6, uk2R, wxoy, p3z3).
• It is theoretically grounded, introducing the first upper bound on fine-tuned LLM safety via KL divergence and STAR scores (qGV6, wxoy, p3z3).
• It demonstrates strong and generalizable empirical results across diverse models, guardrails, and threat scenarios, achieving safety gains without compromising task performance (qGV6, uk2R, wxoy, p3z3).

We hope our responses below address your specific concerns.

Efficient and effective chunking for STAR training

As explained in Sec. 4.1 and Appendix D, STAR scores are computed by querying the guardrail model by appending every $M$ new words in the response, i.e., at positions $\left( x, y_{1:m} \right)$, $\left(x, y_{1:2m}\right)$, ..., where $x$ is the user prompt and $y=\left( y_1, ..., y_T \right)$ is the response sequence. To minimize computation overhead, STAR scores are precomputed and cached along with token positions, enabling construction of a per-token weighting mask that modulates the CE and KL loss terms.

Our STAR-DSS is robust to false positives (FP) and false negatives (FN) chunks

Our STAR-DSS is robust to both FP and FN errors both theoretically and empirically.

Theoretically, as shown in Appendix C, the harmfulness of the finetuned model is provably upper-bounded by two interpretable terms: (1) the guardrail’s chunk-level FN rate at size $M$, and (2) the KL divergence between the finetuned and the reference models. As more accurate guardrails become available, STAR-DSS is expected to benefit directly from better safety estimation.

Empirically, Appendix D.2 shows that smaller chunk sizes $M$ yield finer STAR granularity, which improves safety at the cost of slightly increased preprocessing. We reran experiments with various $M$, each evaluated over 5 seeds, and report the mean ± std below. These results reinforce our choice of $M=5$ as a practical trade-off between robustness and efficiency.

Table A: Smaller chunk size $M$ improves safety by enabling finer-grained STAR scores. LLM: Llama-3.2-1B-Instruct, Guardrail: Granite Guardian-3.1-2B.