POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization

We thank the reviewer for their review and comments.

We will answer the points raised individually.

1. (Other Comments Or Suggestions) I would change the 'toxic question example' in the appendix figure 7 to something a little more explicitly toxic.

We thank the reviewer for this suggestion. In our revision, we will include the following example which is more explicitly toxic. It targets personal privacy.

https://drive.google.com/file/d/1X8yfzFXwMTwxgXzXglaiV8cNAqYIBted/view?usp=sharing

Thank you very much for your valuable time and thoughtful review! We hope you would consider increasing your score if we have addressed your suggestion. Please let us know if you have additional comments and we are happy to follow up. Thanks!

We thank the reviewer for their review and comments.

We will answer the points raised individually.

1. (Claims And Evidence) The slight decrease in safety after POROver could be discussed more explicitly.

We thank the reviewer for this suggestion. We believe the slight decrease in safety may stem from our grid search over the containment threshold parameter. The intermediate values we tested (0.01, 0.03, 0.1) correspond to slightly different points along the safety–usefulness curve. This suggests that our current choices for the containment threshold may be suboptimal, and finer or adaptive tuning could further improve results. In future work, we plan to explore automated methods for optimizing the containment threshold to better balance safety and usefulness.

We will add this discussion to our revised manuscript.

2. Including a wider range of model architectures and sizes for student and teacher models could show the potential generalizability of the proposed approaches.

New student models: We have expanded our analysis to include Falcon-7B as an additional student model, increasing the total number of model families to 3. To further broaden the range of student model sizes evaluated, we also include results for Llama-3.2-11B. Therefore, our analysis now covers student models ranging from 3B to 11B in size. We leave the exploration of models larger than 11B to future work, as we did not have access to sufficient compute resources at this time.

Falcon-7B results: (To be included before April 2nd) Llama-3.2-11B results: (To be included before April 2nd)

New teacher model: We agree that incorporating additional teacher models could provide deeper insight into the generalizability of our methods. Moreover, relying solely on a proprietary teacher model like GPT-4o may limit the broader real-world applicability of our approach and reduce its impact. To address these concerns, we have expanded our analysis to include results using Llama-3-70B, an open-weight model, as the teacher.

Llama-3-70B results:(To be included before April 2nd)

While the exact metric values for the student models vary slightly, all of our main conclusions remain consistent. We believe that including an open-weight teacher model as well as more student models strengthens the robustness of our findings, reduces dependency on proprietary models, and enhances the practical applicability of our methods.

3. While the results are promising, the discussion of the long-term generalization of the proposed methods to real-world tasks could be explored further.

We thank the reviewer for raising this important point. One key aspect of long-term generalization to real-world tasks is robustness to adversarial prompts, as real-world usage often exposes models to unexpected or malicious inputs. To evaluate this, we have added a detailed adversarial robustness analysi. Specifically, we test all of our student models against three adversarial attack methods: Prompt Automatic Iterative Refinement (PAIR), Greedy Coordinate Gradient (GCG), hand-crafted jailbreaks from Jailbreak Chat (JBC), using the behavoral prompts from the JailBreakBench Benchmark.

Jailbreaking results: https://drive.google.com/file/d/13aVd3igdd7JGZ_5PdfQRdcWOkF2Q19UU/view?usp=sharing

For our supervised finetuning approach—i.e., overgeneration with better teacher models—we find that it improves adversarial robustness (measured by attack success rate) against GCG, PAIR and JBC significantly.

We further show that our preference optimization method does not compromise adversarial robustness across any of the three attack types. This demonstrates that it effectively reduces overrefusal without degrading safety under real-world adversarial conditions.

We think that these observations offer strong empirical support for the real-world reliability and generalization of our methods, especially in adversarial and high-stakes deployment settings.

4. (Questions for Authors) How does the computational cost of your approach scale with model size? Do you anticipate any challenges in applying these methods to even larger language models?

In the model sizes we have examined, we found that the training takes longer GPU hours although the convergence times remain similar for the larger models. We have already included a numerical analysis in our original manuscript Appendix D. Other challenges associated with larger models include increased GPU memory requirements and the need for more complex code to enable distributed training. We will discuss these challenges in our revised manuscript.

Thank you very much for your valuable time and thoughtful review! We hope you would consider increasing your score if we have addressed your suggestions and concerns. Please let us know if you have additional comments and we are happy to follow up. Thanks!