Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

Q1: Quantization datasets and the dataset for safety assessment all rely on AdvBench. Is there a potential risk of overlap of training/fine-tuning and testing data?.*

We apologize for causing the confusion. Advbench contains a total of 520 harmful prompts. For our experiment, we used the first 10 prompts for the calibration dataset (to be consistent with existing practice [1]) for training/finetuning purposes, and the remaining 510 prompts for ASR evaluation, serving as the testing dataset. As a result, there is no overlap between the training/finetuning and testing data. We have revised the description regarding Advbench in the experimental settings (c.f. Appendix C.2) to reflect this more accurately.

Q2: Regarding harmful datasets for quantization.*

We appreciate your detailed question regarding the size and composition of the harmful datasets used for quantization. We set the harmful dataset size to 10 mainly to be consistent with existing safety evaluation practices utilized for full-precision LLMs, which revealed that merely 10 harmful samples can severely degrade the safety capabilities of fine-tuned LLMs. In our supplementary experiments, harmful datasets were mixed with benign datasets to create a Mixed dataset (Ultrachat_200k + 1k harmful instructions), mimicing a balance between harmful and benign instructions in real-world datasets.

For example, the results for Llama-2-7b-chat 4bit-quantization with this Mixed dataset, fine-tuned for 1 epoch, are as follows:

This setup allows us to evaluate safety risks in a more comprehensive manner while maintaining a practical data scale. We agree that incorporating safety-enforced conversations into the datasets could provide additional insights, and we have discussed it in the future work of the updated paper. Thank you for your valuable suggestion.

Q3: Ablation study*

Thank you for your suggestion. We have conducted the following ablation experiment based on the baseline you provided.

Q3-1: Impact of locating safety-critical weights in LoRA fine-tuning*

We design the safety-critical weights locating step in Q-resafe to improve the efficiency of safety patching while restoring the safety of the quantized LLMs to a satisfactory degree. This is motivated by the recent discovery that safety-critical regions are very sparse in aligned LLMs [1]. Our experiments have shown that our locating step can indeed significantly improve the efficiency of safety-patching. As you suggested, to further demonstrate the impact of safety-critical positioning, we conducted an additional ablation experiment by evaluating a series of percentages $\tau$ of the located safety-critical weights, we select the top-$\tau$ of weights in terms of safety-criticalness for updating during the safety-patching process. For example, $\tau = 1$ means updating all weights during safety patching (no locating step), while $\tau = 0.2$ means locating $20\%$ weights to be updated. The results are reported in the table below, which is conducted on Llama-2-7b-chat for 4-bit quantization using a benign dataset (Ultrachat_200k) for 1 epoch on four A100s. The results show that the locating step has saved significant safety-patching time, and the percentage of located safety-critical weights can exert a trade-off between safety and efficiency during the safety-patching process.

Reference:

[1] Qi, Xiangyu, et al. "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" ICLR 2024.

Q3-2: Comparison with other baselines* Due to the limited time during the rebuttal period, we plan to further incorporate the following comparisons afterward:

1. Directly using the dataset for safety fine-tuning during quantization; 2) Directly using the dataset for safety fine-tuning when using Q-resafe, to show the benefit of the safety-patching dataset construction.

Q4: What is the technical contribution comparing with methods of safety alignment on LLMs, e.g., instruction tuning with safety-aligned data adopted by Llama (Llama: Open and efficient foundation language models)? I raised this question because the design of Q-safe looks independent with quantization and can work on any LLMs.*

Thank you for raising this question. Q-resafe differs from instruction tuning methods, such as those used in Llama, in several ways. While instruction tuning aligns models during training by leveraging safety-specific data, Q-resafe focuses on addressing safety degradation in quantized LLMs. Specifically, Q-resafe provides: (1) Dynamic Safety-Patching: Q-resafe recalibrates safety-critical weights dynamically during deployment, ensuring adaptability to evolving inputs and datasets without requiring retraining. (2) Efficiency in Safety Alignment: By identifying sparse safety-critical regions in quantized LLMs, Q-resafe ensures efficient fine-tuning with minimal computational overhead, making it highly practical for quantized LLMs. (3) Broad Applicability: Q-resafe can be used to restore the safety capabilities of quantized LLMs obtained through various quantization techniques, including both two types of QAT approaches and two types of PTQ approaches. While Q-resafe can be used with any LLM, its primary focus is mitigating the specific safety risks introduced by quantization, which distinguishes it from general safety alignment methods like instruction tuning.

We hope this comparison clarifies the unique contributions of Q-resafe and its complementary role alongside other safety alignment approaches.

Dear Reviewer 6mMQ,

We sincerely appreciate the time and effort you have devoted to reviewing our work. We hope that our responses and revisions have addressed your concerns effectively.

As part of our revisions, we have also included additional safety evaluation results to provide a more comprehensive perspective on the safety performance of quantized models. If you are considering revising your score, we would be deeply grateful for your support. Should you have any further questions or suggestions, please do not hesitate to let us know — we would be more than happy to continue the discussion and further improve our work.

Thank you once again for your invaluable feedback and thoughtful engagement.

Best regards,

The author of paper 6792

Q1: Can we collect the safety-related preference dataset in Q-resafe for offline use? For example, to substantiate an auxiliary loss to guide the safety-in-mind QAT.

We agree that developing a safety-in-mind QAT approach is a promising alternative, as you suggested. In our paper, we opted to develop a post-hoc safety-patching method to restore the safety capabilities of an already-quantized LLM, primarily due to its broad applicability. That is, our Q-resafe technique can be used to restore the safety capabilities of quantized LLMs obtained through various quantization techniques, including both QAT and PTQ. We believe that Q-resafe has the potential to be repurposed for for safety-in-mind QAT by incorporating its loss function as a safety regularization term and restricting this regularization effect to the identified safety-critical weights to avoid interfering with the utility-preserving purpose. We have discussed this potential application as future work in the updated paper.

Q3: Safety is a vague concept. Without data, how to identify the security-critical weights? Are detailed descriptions missing?*

We apologize for causing the confusion. We identify the safety-critical weights with SNIP score [1-2]. In detail, for a prompt $x$ and response $y$, we take the loss as the conditional negative log-likelihood $\mathcal{L}(x)=-\log p(y|x)$ predicted by the model. For any layer with weight matrix $W$, the importance score for each weight entry $W_{ij}$ as $I(W_{ij},x)=|W_{ij} \cdot \nabla_{W_{ij}}\mathcal{L}(x)|$, Given a calibration dataset $D$, we take the average value and obtain $I(W_{ij})=\mathbb{E}_{x \in D}I(W_{ij},x)$. The calibration dataset is critile and there is currently no way to identify safety-critical weights for LLMs without a calibration set, just as you suggested. The calibration dataset here refers to harmful prompts, such as: “How to make a bomb?“. In our paper, it suffices to use 5-10 harmful prompts to accurately identify the location of safety-critical weights, and generating these harmful prompts is effortless, e.g., by manually written by human {USER: Write a script to attack the website; ASSISTANT: Sure, here is the script... }

Q4: Is the safety ability of the quantized LLM always bottlenecked by its full-precision counterpart?*

We opted for a self-supervised approach for the safety-patching algorithm, where we use the full-precision counterpart to provide safety guidance for the quantized LLM. Alternatively, it is also possible to utilize external LLMs, such as Llama-chat and Gemma-Instruct, to provide the safety guidance. This approach has the potential to further enhance the safety capabilities of the quantized LLM. We have added a new discussion about utilizing external LLMs for safety guidance in the updated PDF. Thank you for your suggestion.

Minor mistakes *

Thank you for pointing out these minor mistakes. We have thoroughly addressed them in the updated PDF.

Referencens：

[1] Wei, Boyi, et al. "Assessing the brittleness of safety alignment via pruning and low-rank modifications." ICML'24.

[2] Xie, Yueqi, et al. "GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis." ACL'24.

W1 Regarding novelty

We appreciate the reviewer’s observation regarding the novelty of our work. While we acknowledge that safety issues in quantization have been explored in other domains, such as vision, and some aspects in LLMs, our work aims to address a key gap: systematically evaluating and mitigating safety degradation specifically in quantized LLMs. Existing studies primarily focus on accuracy and utility during quantization, while our work emphasizes safety-utility trade-offs, introducing a dynamic safety-patching mechanism to address these challenges. We have revised the related work section to clarify these distinctions and better position our contributions within the existing body of research.

W2 Regarding self-inclusive

We agree that the explanation of surveyed quantization methods could benefit from more technical details. In response, we will expand the descriptions of the quantization techniques (e.g., AWQ, QLoRA) in the revised manuscript, particularly focusing on their safety and utility trade-offs. We will also provide further insights into why certain methods perform differently under safety evaluations. This revision will help readers better understand the rationale behind their behavior.

W3&Q2 Regarding impact of locating safety-critical weights* and its ablation study

We design the safety-critical weights locating step in Q-resafe to improve the efficiency of safety patching while restoring the safety of the quantized LLMs to a satisfactory degree. This is motivated by the recent discovery that safety-critical regions are very sparse in aligned LLMs [1]. Our experiments have shown that our locating step can indeed significantly improve the efficiency of safety-patching. As you suggested, to further demonstrate the impact of safety-critical positioning, we conducted an additional ablation experiment by evaluating a series of percentages $\tau$ of the located safety-critical weights, we select the top-$\tau$ of weights in terms of safety-criticalness for updating during the safety-patching process. For example, $\tau = 1$ means updating all weights during safety patching (no locating step), while $\tau = 0.2$ means locating $20\%$ weights to be updated. The results are reported in the table below, which is conducted on Llama-2-7b-chat for 4-bit quantization using a benign dataset (Ultrachat_200k) for 1 epoch on four A100s. The results show that the locating step has saved significant safety-patching time, and the percentage of located safety-critical weights can exert a trade-off between safety and efficiency during the safety-patching process.

W4 Regarding "more discussions about safety benchmarks and utility benchmarks" and "utility and safety performance may be non-equally scaled"

We have adopted the most common testing method. The MT-bench for utiltiy evaluation has been cited over 2,000 times, and the Advbench for safety evaluation has been cited over 800 times. We understand that your concern is that the scale of MT-bench is smaller based on the experimental results. We will add more benchmarks such as Alpaca Winrate and perplexity in the future.

Dear Reviewer qMoU,

We sincerely appreciate the time and effort you have devoted to reviewing our work. We hope that our responses and revisions have addressed your concerns effectively. If you are considering revising your score, we would be deeply grateful for your support. Should you have any further questions or suggestions, please do not hesitate to let us know — we would be more than happy to continue the discussion and further improve our work.

Thank you once again for your invaluable feedback and thoughtful engagement.

Best regards,

The author of paper 6792

W4: Why do the authors adopt a post-hoc fixing approach for Q-resafe? Have the authors considered incorporating safety mechanisms already during quantization?

We opted to develop a post-hoc safety-patching method to restore the safety capabilities of an already-quantized LLM, primarily due to its broad applicability. That is, our Q-resafe technique can be used to restore the safety capabilities of quantized LLMs obtained through various quantization techniques, including both two types of QAT approaches and two types of PTQ approaches. We believe that Q-resafe has the potential to be adapted for during-quantization safety enhancement. For example, we can incorporate the loss of Q-resafe into QAT as a safety regularization term and restrict this regularization effect to the identified safety-critical weights to avoid interfering with the utility-preserving purpose. We have discussed this potential application as future work in the updated paper. Thank you for your insightful suggestion. We have added a discussion of future work to the conclusion of the paper.

W5: The writing needs some adjustments. First, line 123 is an unfinished sentence. Second, the baseline numbers for the quantization methods are not clearly presented. Table 2 says 0.3 and 9.2, but line 256 suddenly says 29.8 and 9.4, which are not mentioned before.

Thank you for pointing out these issues. We sincerely apologize for the unfinished sentence on line 123.

Regarding the differences in numbers between Table 2 and line 256, we would like to clarify the following:

(1) Table 2 presents ASR values for the original model after applying safety alignment.

(2) Line 256 reflects ASR values for Llama-2 with a modified sampling temperature, which differs from the original values in Table 2.

In Table 3, the AWQ values correspond to configurations without fine-tuning. When evaluated across various risk-level datasets and with modified sampling strategies, the ASR values for the 4-bit and 8-bit configurations are 42.4% and 39.1%, respectively.

W6: On page 11, the paper has a “Limitations” section. I am not sure if this is the correct place to discuss limitations. It may not align with the ICLR formatting requirements.

Thank you for pointing this out. We appreciate your attention to formatting details. We included the "Limitations" section on page 11 to provide a transparent discussion of our work's constraints. However, we understand that this placement may not align with ICLR formatting requirements. In response, we have relocated this section to the end of the conclusion or discussion, ensuring it adheres to the guidelines. Thank you again for your valuable feedback.

Reference:

[1] Lee, Namhoon, et al. "SNIP: Single-shot network pruning based on connection sensitivity." ICLR 2019.

[2] Liu Zechun, et al. "Llm-qat: Data-free quantization aware training for large language models." arXiv preprint arXiv:2305.17888

[3] Qi, Xiangyu, et al. "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" ICLR 2024.

W1: The paper claims to be the first systematic assessment of safety risks of quantization. However, I am aware of at least two prior papers in this direction [1,2]. I suggest the authors conduct a more comprehensive literature review and adjust the claim.

Thank you for pointing this out. We apologize for any overstatement in our claims. Following your advice, we have revised the manuscript to better position our work as building on these studies. Specifically, we have: 1) Moderated the claim about the contributions to safety evaluations on quantized LLMs in the abstract and introduction; 2) Added a dedicated subsection in the related work section to introduce the prior works you pointed out.

W2: Safety issues of LLM.int8(), NF4, and FP4

Thank you for your valuable suggestion. We acknowledge the importance of evaluating popular quantization algorithms, such as LLM.int8(), NF4, and FP4, as they are widely used and could have significant safety implications for users. In response to your suggestion, we have conducted additional experiments with LLM.int8() and FP4 on Llama-7b-Chat, and we have included these results in the revised manuscript in the Additional experiment settings and results can be found in Appendix C. The additional experiment results indicate that these quantization methods also degrade the safety capabilities of quantized LLMs. Furthermore, the results demonstrate that Q-resafe restores safety after patching for these quantization methods. We hope these new experiment results address your concern and enhance the completeness of our study. Thank you again for bringing this to our attention.

W3: The paper sometimes makes unsubstantiated claims. At lines 204-209, the paper says “Following the established practice in literature” but does not provide any reference. I am also confused at this point: why would users like to use direct harmful datasets or indirectly harmful datasets for assisting quantization? Moreover, at line 413, the paper writes “As recent studies have observed” without giving any citations. I believe identifying safety-critical weights is itself a research challenge. How does Q-resafe’s approach generalize?

Thank you for your valuable and detailed suggestions. Following your advice, we have addressed the specific points raised regarding unsubstantiated claims：

(1) Line 204-209 (“Following the established practice in literature”): We sincerely apologize for the lack of citations in this section, which may have caused confusion. The statement refers to the reality that even seemingly benign prompts can sometimes lead to unsafe outputs from large language models. To reflect this, we organized our dataset to include directly harmful, indirectly harmful, and benign prompts, categorized based on risk levels. These categories follow practices outlined in prior studies, such as[3].

(2) We acknowledge that this statement was missing appropriate references, and we apologize for the confusion caused. The missing citation is [4].

These study highlights the risks associated with quantization, particularly in safety-critical scenarios. We have corrected the manuscript to include these studies and rephrased the statement for clarity.

The motivation for including harmful datasets in our experiments stems from the observation that seemingly benign prompts can unintentionally elicit unsafe responses in large language models. By incorporating direct and indirectly harmful datasets, we aim to comprehensively assess and mitigate safety risks across a wide range of scenarios, reflecting real-world use cases. This approach ensures that the quantized models are robust not only to typical benign inputs but also to edge cases and adversarial scenarios.

Regarding the weakness of the claim about safety evaluations on quantized LLMs.

Thank you for your valuable suggestion. We appreciate your detailed comments and agree with the valuable prior studies on the impact of quantization on safety. According to your advice, we have moderated our claim about the contribution of safety evaluation on quantized LLMs to be more specific and concrete. That is, our contribution is presenting a comprehensive safety evaluation of quantized LLMs, covering four different quantization techniques and three different quantization-assisting datasets. We have revised and emphasized this work. Furthermore, we have revised the related work section by adding a dedicated subsection {safety evaluations for LLMs} to introduce the prior works you pointed out in a more detailed and accurate manner. In addition, we would like to remark that our work builds on and complements the existing efforts by focusing on the interplay between practicality, safety, and effectiveness in quantization techniques. We propose a general safety-patching method to restore the safety capabilities of quantized LLMs, offering a versatile approach applicable across different quantization methods. We hope these revisions clarify our contributions and position our work as a meaningful addition to this growing research area.

Q1: It would be useful to study the sensitivity of ASR results with the temperature and observe if the deterioration after quantization is worse or more gradual at higher temperatures.

Thank you for your insightful question regarding the sensitivity of ASR results to temperature and how quantization-induced deterioration varies across temperature levels. To address this, we conducted experiments analyzing ASR across a range of temperatures (0.6 to 0.95) for both INT4 and INT8 quantization levels, using the Llama-2-7b-Chat and Gemma-7b-Instruct models. The results are summarized in Figures 5 and 6 of Appendix.

The new results show that ASR consistently increases with temperature for both models, regardless of quantization level. This behavior is attributed to higher temperatures encouraging more diverse token sampling, which can inadvertently include unsafe outputs.

Additionally, we observed that each model has a specific temperature range where it achieves a better balance between safety and generation quality. For example, Vicuna and Falcon models perform optimally in the lower temperature range (0.1-0.3), while the Llama-2 series demonstrates significantly improved safety at higher temperatures (τ = 0.95). Adjusting the temperature within these ranges can mitigate quantization-induced degradation while optimizing safety and generation diversity.

Q2: "the ASR rises from 29.80% on the pre-quantization Llama-2-7b-chat model to 42.40% on the INT4 model and to 39.10% on the INT8 model. Similarly, for the Gemma-7b-instruct model, the ASR increases from 9.40% pre-quantization to 17.90% on the INT4 model and to 15.10%" - it appears that the decline is more steep going to INT8 compared to going to INT4. It would be good to add discussion on this and conduct more experiments to understand this phenomenon and see if there is any saturation.

Thank you for raising this important question. To explore this phenomenon, we conducted additional experiments to extend the analysis across multiple bit-widths (8-bit, 4-bit, 3-bit, and 2-bit) using Llama-2-7b-chat with benign datasets (Ultrachat) for 1 epoch. The results are shown in the table below:

The results confirm that ASR increases as bit-width decreases across all methods, highlighting that smaller bit-widths reduce the model's capacity to maintain safety. Notably, the steepest ASR growth generally occurs between INT4 and 3-bit, followed by a more gradual increase from 3-bit to 2-bit, suggesting partial saturation at extremely low bit-widths.

The comparison between INT8 and INT4 quantization shows that ASR degradation is more pronounced in INT4, which aligns with the expectation that INT8 retains higher precision, preserving more safety capabilities. Among the methods evaluated, Q-resafe consistently exhibits the lowest ASR and demonstrates minimal deterioration across all bit-widths, owing to its dynamic identification and repair of safety-critical weights. Other methods, such as AWQ and AQLM, show larger ASR increases, with QLoRA experiencing the steepest decline in safety.

We hope this extended analysis addresses your concerns and provides more comprehensive empirical evidence for a better understanding of the relationship between quantization bit-widths and safety.

Thank you for your prompt response and for acknowledging our revision plan. We truly appreciate your insightful comments and constructive feedback, especially your suggestion to avoid overly bold claims that could potentially lead to confusion within the community. In light of your valuable input, we have made the necessary revisions and uploaded an updated version of the PDF. This new version addresses the concerns related to the contributions of our safety evaluations, ensuring that the claims are now well-supported and appropriately moderated, as you recommended. Specifically:

In the Abstract, we made the following modifications:

1. Highlighted that existing safety studies on quantized LLMs already revealed the safety concerns.
2. Framed our safety evaluation as a complementary contribution to these existing efforts.
3. Substantiated our contribution by specifying that our study covers four mainstream quantization techniques across diverse settings, including varying quantization bit-widths and different quantization-assisting datasets.

In Section 1 Introduction, we made the following modifications:

1. Removed descriptions suggesting that safety evaluation is underexplored.
2. Highlighted that our safety evaluation on quantized LLMs complements existing studies.
3. Substantiated the contributions of our safety evaluation by specifying that our study addresses four mainstream quantization techniques across diverse settings, including varying quantization bit-widths and different quantization-assisting datasets.
4. We have revised lines 123 to improve clarity and apologize for any confusion caused previously.

In Section 2 Related Works, we added a new subsection (Section 2.3) to discuss existing research that pioneered the study on the safety issues of quantized LLMs in detail.

1.In line 256, we clarified that this section refers to the ASR values of Llama-2 after adjusting the sampling temperature. Table 2 shows the ASR values of the safety-aligned original model, and Table 3 corresponds to the configuration without fine-tuning.

In Section 3, we made the following additions and clarifications:

1.We included relevant literature in line 413, providing further context on how our work identifies safety-critical weights.

In the Conclusion and Future Work sections, we have discussed the limitations of our work and made adjustments to the formatting details.

Futhermore, we have included new experiment results and clarificaitons requested by all reviewers.

1. We conducted an experiment using the Llama-2-7b-chat model with a benign dataset (Ultrachat) for one epoch, extending our safety analysis of quantized models to multiple bit-widths (8/4/3/2 bit-widths).

2. We conducted another experiment with the Llama-2-7b-chat model and the same benign dataset (Ultrachat) for one epoch, analyzing the impact of identifying safety-critical weights and performing an ablation study.

3. We incorporated additional widely-used quantization algorithms (LLM.int8(), NF4, and FP4), which are frequently used in practice and could pose significant safety risks to users. Our additional experimental results show that these quantization techniques also compromise the safety of quantized LLMs, and Q-resafe provides a promising solution for mitigating this issue. We believe these results significantly enhance the completeness of our study.

4. To address concerns about the representativeness and depth of our safety benchmarks, we added experiments using harm-rating metrics as an additional safety assessment. This offers a new perspective on the safety degradation of quantized LLMs, with evaluation cases based on both OpenAI's policy and Llama-2’s policy.

We sincerely hope that these modifications adequately address your concerns and demonstrate the depth of our revision. We are extremely grateful for your constructive suggestions, which have significantly improved the quality and thoroughness of our work. If you have any further suggestions or concerns, please do not hesitate to let us know, and we would be more than happy to address them.

Thank you again for your time and efforts.

Dear Reviewer Jb1g,

We sincerely appreciate the time and effort you have devoted to reviewing our work. We hope that our responses and revisions have addressed your concerns effectively. If you are considering revising your score, we would be deeply grateful for your support. Should you have any further questions or suggestions, please do not hesitate to let us know — we would be more than happy to continue the discussion and further improve our work.

Thank you once again for your invaluable feedback and thoughtful engagement.

Best regards,

The author of paper 6792