Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning

Thank you for your insightful feedback. We have provided our responses below, and we hope they clarify the points you raised. If our responses have adequately addressed your initial concerns, we would be grateful if you would consider adjusting your evaluation accordingly.

(A)

We define pseudo-harmful prompts in line 39 of the Introduction- they are queries which appear potentially harmful but are benign in nature.

(B)

We address the question of generalization to unseen datasets through our experimental setup. We train our model on 25 examples each from XSTest, Scope, ORBench, and PHTest, and then evaluate on the corresponding held-out partitions of those datasets. In addition, we include OKTest as an entirely unseen benchmark to assess how well our learned shift extends beyond the trained domains. The results are summarized in Table 1.

To further address concerns about out-of-domain generalization, we also conduct a supplementary experiment in which training is restricted to only 25 queries from XSTest and ORBench along with the harmful and harmless queries from Hexphi and UltraChat. As shown in the experiments below, the model retains its out-of-distribution (OOD) robustness even when we vary the over-refusal distributions used during training.

(C)

We agree that broader comparisons can further strengthen our work. We have now incorporated comparisons against multiple baselines. First, we present the following table discussing the features of all existing approaches and their comparison with ACTOR.

1. Results for Llama-2-7b-chat-hf

2. Results for Gemma-7b-it

Our results continue to show that ACTOR outperforms these broader baselines in reducing over-refusals while maintaining safety scores.

(D)

We acknowledge that our approach relies on access to model internals, which may be unavailable for users of closed-source or black-box systems. However, we do not view this as a fundamental limitation. In practice, closed-source model developers do have full access to these internals and can adopt our method if they choose. Moreover, the ability to examine the model’s internal representation space provides critical insights for understanding and mitigating over-refusals—insights that purely API-level or prompt-based methods cannot readily capture.

Thank you for your insightful feedback. We have provided our responses below, and we hope they clarify the points you raised. If our responses have adequately addressed your initial concerns, we would be grateful if you would consider adjusting your evaluation accordingly.

(A)

Our approach differs fundamentally from the single‐vector “surgical” method [1], which computes a “false‐refusal” vector from a set of harmless, harmful, and pseudo‐harmful queries. We observe that simply switching the pseudo‐harmful dataset from ORB-H to OKTest led to substantial fluctuations in performance.

These fluctuations arise because the method bakes a single fixed vector directly into the model’s weights and applies it uniformly to every query—a “one‐size‐fits‐all” mechanism. Consequently, the model’s efficacy heavily depends on the specific distribution from which the vector is extracted, rendering it brittle under distributional shifts. shifts. We explore a natural extension of this approach in the fine‐tuning setting under the subsubsection “Would Uniform Shifts Work?” (Sec 3.2). We show that a single uniform vector shift leads to destructive fine‐tuning, with more discussion in Appendix B.

Instead of a single fixed vector, ACTOR repeatedly updates both the model parameters and the refusal direction during fine‐tuning. Mathematically, rather than enforcing a constant scaling factor, ACTOR’s loss function promotes an “individualized” or “just enough” shift—proportional to each query’s projection onto the refusal direction—so that the model can adapt its internal representation on a query‐by‐query basis via minimal intervention. We discuss these design choices in more detail in Section 3. This iterative, dynamic mechanism makes ACTOR robust: instead of relying on a single axis of correction, it exploits the full capacity of the model’s internal activation space to handle diverse data distributions.

To further validate this robustness, we computed the initial refusal direction for ACTOR using three different harmful data distributions as shown in Fig 5 (Sec 4.2) and the table below. ACTOR consistently maintains strong performance, even when the refusal direction is computed from different data sources.

We hope this clarifies why our method is not merely an extension of [1] augmented with fine‐tuning. Our design deliberately addresses the pitfalls of a single‐vector solution and provides a more adaptive, reliable alignment strategy.

(B)

Baselines- Kindly refer to our rebuttal to Reviewer eEU9 (C). Both SafeDecoding and DRO suffer from a major over-refusal problem and perform worse than SFT.

(C)

The 210 benign examples are randomly sampled from the 7 categories of the UltraChat Dataset (n=30). The 25 queries from XSTest, SCOPE, OR-Bench-Hard-1k, PhTest benchmarks are also sampled randomly with their held-out versions used for evaluation and OKTest serving as an OOD dataset.

We experiment with 2 additional settings where we set n=15 and 50 for UltraChat-

Similarly, we also conducted additional ablation experiments by selecting 10 and 50 random over-refusal queries from the abovementioned datasets-

We also want to highlight that our method remains effective even under low data budgets. As shown in Fig 4, using only 25 over‐refusal queries during training yields performance gains that surpass SFT trained on 100 over‐refusal queries, underscoring the data efficiency of our approach.

(D)

We agree that adding analysis on specific categories would make the paper more rich. XSTest, ORBench-Hard-1k and SCOPE are the benchmarks with such categories included. We show the Compliance Rates on these categories before and after intervention with ACTOR here.

[1] Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation

We appreciate your thoughtful remarks and the positive rating you assigned to our paper. Below, you'll find our responses, which we hope clarify the points you raised.

(A)

While multi‐turn attacks indeed pose a more realistic challenge, there are currently no established benchmarks specifically designed for multi‐turn over‐refusal scenarios. Evaluating both multi‐turn over-refusal and multi‐turn safety, as well as understanding how they intersect, remains an open research problem. We therefore leave this exploration to future work, and we appreciate your suggestion to further examine the robustness of our method in more complex dialog settings.

(B)

We found that existing classifiers such as HarmBench, LlamaGuard, and WildGuard can be overly conservative for the pseudo-harmful queries used in our benchmarks, often flagging them as harmful. For example- LlamaGuard classifies 33% of the total queries from ORBench-Hard-1k as harmful.

Earlier approaches [1][2] in our line of work include human evaluation and string matching for evaluation. To achieve a balance between context‐awareness, reproducibility, and scalability we opted for an LLM judge with a carefully crafted system prompt—an approach used extensively in related literature [3]. We provide all relevant prompts and details in Appendix E, allowing others to replicate our setup and compare results more directly.

(C)

Since our current approach only fine-tunes a single layer—already providing a relatively efficient setup—we consider integrating LoRA into our design as an exciting avenue for further increasing parameter efficiency and performance even on larger models.

Thanks again for the valuable suggestions.

[1] Cao, Zouying, Yifei Yang, and Hai Zhao. "Nothing in excess: Mitigating the exaggerated safety for llms via safety-conscious activation steering." arXiv preprint arXiv:2408.11491 (2024).

[2] Wang, Xinpeng, et al. "Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation." arXiv preprint arXiv:2410.03415 (2024).

[3] Qi, Xiangyu, et al. "Fine-tuning aligned language models compromises safety, even when users do not intend to!." arXiv preprint arXiv:2310.03693 (2023).

Thank you for your thoughtful feedback. Below, you'll find our detailed responses, which we hope clarify the points you raised.

(A)

We appreciate the opportunity to clarify the basis of our claims in Sec 2. While we had experimental results (Sec 4) that support these points, we realize they were not explicitly cited. We will revise the manuscript to better link these findings to our claims.

1. We claim that SCANS and the “surgical” method [1]—are highly sensitive to the specific datasets used to derive their refusal vectors. In Figure 5, we show that SCANS exhibits significant performance fluctuations when we vary the harmful datasets used. Likewise [1] also exhibits similar variance when switching among different pseudo-harmful datasets.

2. We also claimed that SCANS and [1] offer a uniform, one-size-fits-all adjustment to the model, applying the same refusal vector—regardless of each query’s unique activation patterns. This approach can lead to unnecessary or insufficient corrections, as each query requires a different level of ablation to ensure compliance (Sec 3.2).

ACTOR’s Adaptive Mechanism: In contrast, our proposed approach employs a dynamic training objective- moving ahead from specific task vectors. ACTOR repeatedly updates both the model parameters and the refusal direction during fine-tuning, enabling an input-dependent shift proportional to each query’s projection onto the refusal direction. This underpins ACTOR’s robustness; it is not limited to a single axis correction but leverages the full capacity of the model’s internal activation space to maintain robust performance even under distributional shifts (Fig 5). As shown by our experimental results in the paper (Tables 1–4) and the addition of new baselines (Reviewer eEU9 response (C))- ACTOR outperforms all baselines in balancing both compliance and safety. We have also included a table comparing existing methods with ACTOR to further support our claims on their limitations in our response to Reviewer eEU9 (C).

(B)

We would like to clarify that ACTOR is distinct from ReFT in both design and motivation.

1. Parametrization: Rather than introducing additional parameters, ACTOR fine-tunes a single layer of the original model, avoiding the overhead associated with auxiliary interventions.

2. Learning Objective: Its objective departs from standard output-based losses by encouraging a “just enough” shift in the model’s internal representations, scaled by each query’s specific projection onto the refusal direction. This prevents both excessive and insufficient corrections, which can result when a one-size-fits-all approach is applied.

3. Annotated Data: Unlike ReFT, ACTOR does not rely on full response supervision as it draws on internal activation information as its supervision signal making it more cost-effective and straightforward to deploy.

Motivationally, ACTOR is designed for cases where the degree of over-refusal varies by query, requiring individualized corrections (Sec 3.2). This focus drives both the training algorithm and the learning objective, ensuring a reduction in over-refusal while being computationally light. We will revise the appropriate sections to highlight these distinctions and justify our novelty.

(C)

We would like to clarify that the intent of the theoretical analysis is not to explain why the “Just Enough” Shift preserves natural performance or why fine-tuning a single layer is sufficient. Rather, as stated in Lines 231–233, the theory is designed to offer intuition behind the design of our training objective—specifically, why subtracting out the projected activation shift with the refusal direction enables targeted correction for over-refusal, without imposing a uniform change across all inputs.

We agree that theory is simplified to maintain clarity and provide a conceptual foundation for the method’s core ideas. It is not intended to serve as a comprehensive performance guarantee.

As for the claims regarding natural performance preservation and single-layer sufficiency, we support these empirically through extensive experiments in Sec 4.2.

1. The “Just Enough” Shift mechanism maintains high performance on benign queries (Table 1)

2. Fine-tuning a single layer already yields substantial improvement in over-refusal mitigation without degrading natural performance (Table 3). We will revise the text to clarify the scope and role of the theoretical analysis and explicitly distinguish it from our empirical findings.

(D)

Comparison With Baselines: Kindly refer to our rebuttal to Reviewer eEU9 (C)

Ablation Study on the Fine-Tuning Layer: An ablation on the choice of the fine-tuning layer is present in Appendix C.1

[1] Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation