Exploiting LLM Quantization

We would like to thank the reviewer for their time and effort spent reviewing, their insightful comments, and for their overall positive assessment of our work. Below, we address the reviewer’s questions and comments.

Q1: Do your findings generalize also to other popular LLMs?

Yes, as already indicated by our results on 5 LLMs with varying sizes, the presented safety threat is largely independent of the targeted LLM. To further underline this, we extend our evaluation to the popular Mistral-7B and Llama 3.0-8B models, targeting the content injection scenario in the scope of this rebuttal:

Mistral-7b

Llama 3.0-8b

As we can observe from the above table, our attack is effective also against these LLMs. Note that due to resource limitations, we are unfortunately unable to run attacks on Mixtral-8x7B or larger models, however, as we have shown that our attack can scale from 1B LLMs to 8B LLMs, we believe that further scaling would also be possible.

Q2: Are optimization-based quantization methods, such as GPTQ or AWQ captured by the presented threat model?

No, such compute-intensive optimization-based quantization methods are not covered by our current threat model. We assume that the target victim users do not possess enough compute to even run full-precision model inferences, much less so to calibrate quantizations such as GPTQ or AWQ. Note that GPTQ or AWQ quantized models are usually constructed and calibrated by a third party and distributed already in quantized form; reflecting a fundamentally different mechanism from the focus of this paper. We target the popular scenario where the users download full-precision models and quantize them locally using low-resource zero-shot methods.

Nonetheless, we agree with the reviewer that examining if the presented threat extends to the mentioned optimization-based quantization methods is an interesting and crucial future work item, potentially raising further serious safety concerns.

Q3: What would be the impact of this exploit on aligned LLMs?

We believe that the fact that the model has been safety tuned before would not change the impact of the attack. We have two reasons to believe so: (1) the attacker conducts full finetuning of the base model to inject the malicious behavior, which has been shown to remove the impact of the safety alignment [1], and (2) the injected malicious behavior does not even have to be something that is originally covered by the safety training, for instance our scenarios of insecure code generation or content injection.

To quantitatively demonstrate this, we additionally conducted a content injection experiment on Phi-3-Instruct, an aligned chat model.

Despite the alignment training that this model went through, our attack pipeline is still successful, creating a clear contrast between the full-precision and quantized models.

Phi-3-mini-4k-instruct

References

[1] X Qi et al. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. ICLR 2024.

First, we would like to thank the reviewer for their efforts spent reviewing our paper and for their highly positive assessment. We address the reviewer’s questions and comments below.

Q1: Can you please clarify the threat model?

Certainly. The threat model assumes an attacker that either downloads an LLM available on Hugging Face (e.g., official model releases), or obtains a benign LLM through other channels. Then, the attacker injects their attack into this model using the local attack procedure described in Section 3. This will provide the attacker with an LLM that is benign in full-precision and malicious when quantized. Then, the attacker uploads this LLM to a model sharing hub, such as Hugging Face, where it is evaluated in full-precision and is available for anyone to download and use locally. Additionally, the resulting full-precision LLM may exhibit properties that make it attractive for other users to download, as demonstrated in step 2 in Figure 1 (e.g., high code security rate in full-precision, as in the first evaluated attack scenario). Note here that the attacker does not necessarily want to deceive users into thinking that their attacked model is a factory model (however they may do so, by doing for instance typo-hogging). It may well be that the attacker simply uploads the attacked model as a specialized fine-tune of a factory model, fully in their own name, as done frequently on Hugging Face. We will clarify the corresponding passages in the next revision of the paper.

Q2: Would running safety evaluation on the quantized models defend against the presented threat?

Yes, comprehensively red-teaming the quantized model would enable one to uncover the vulnerability. Although, please note that, as always with red-teaming, finding the vulnerability would not be guaranteed. Nonetheless, we believe that this is in fact one of the key messages of the paper; highlighting that (1) current safety evaluation practices only looking at the full-precision model are insufficient, and (2) current practices in evaluating quantization methods only looking at model perplexity and benchmark performance lack safety considerations. We will make sure to highlight these points more clearly and prominently in the next revision of the paper.

Q3: Can you please clarify the metrics used in the experiments?

Definitely, and we will make sure to present them more clearly in the next revision of the paper:

Secure code generation scenario: Following [1], the percentage of code completions without security vulnerabilities measured using a state-of-the-art static analyzer, CodeQL.

Over-Refusal scenario Following [2], the percentage of responses by the model to queries from a subset of the databricks-15k dataset that the model refuses to answer citing plausible sounding reasons (“informative refusal”), as judged by GPT-4.

Content Injection scenario Following [2], the percentage of responses to queries from a subset of the databricks-15k dataset that contain the target word “McDonald’s”.

Note that Appendix A.1 contains further details on the experimental setup, including details on the employed metrics.

References

[1] J He & M Vechev. Large language models for code: Security hardening and adversarial testing. CCS 2023.

[2] M Shu et al. On the Exploitability of Instruction Tuning. NeurIPS 2023.

First and foremost, we would like to thank the reviewer for their insightful review and their positive assessment of our paper. We especially appreciate that the reviewer shares our view about the importance and relevance of the demonstrated exploit.

In response to the reviewer’s concern about the main contributions of this paper; we believe that beyond the indeed significant time reduction our attack brings, our key contributions are (1) demonstrating the feasibility of such a quantization exploit for the first time on LLMs, where the attacked quantization schemes are widely deployed; (2) pointing out and highlighting the relevance of the demonstrated exploit in the context of current model sharing practices; and (3) demonstrating the vast diversity of malicious behaviors enabled by the attack, constituting a paradigm shift in the severity of the proposed threat when compared to its relevance for simple image classifiers.

Below, we address the reviewer’s remaining questions and comments.

Q1: Could you please include the results of the original LLM when quantized?

We agree with the reviewer that this would be an insightful addition to our result tables, which we omitted in the submitted version as this would have increased the table sizes by three rows for each examined model and scenario. Further, the quantized results on the original model are fairly close to those on the unquantized model, as the table below also demonstrates.

Performance of quantized original models

Following the reviewer’s advice, we will include the full quantized original model results in the appendix of the paper, and include a qualitative description of those in an early part of the experimental section.

Q2: Can you please elaborate on how step (3) of the attack is carried out?

Certainly. In step (3) of the attack, given the constraints obtained in step (2) that ensure that the resulting model quantizes to the same malicious model as the original malicious model, the attacker performs PGD training. Here, depending on what the attack objective was, the attacker performs training that goes “against” the training objective of the malicious first step. For example, in the scenario of secure code generation, in step (1) the attacker first trains a model to generate insecure code. Then, in step (3), using the same training pipeline, but this time swapping the secure and insecure examples, the attacker trains the model to generate secure code. Note that while this training ensures that the resulting model still quantizes to the same malicious model as before, it does not guarantee that we find a benign model. However, as demonstrated by our experiments, empirically this is possible on real-world production models without any further care or tricks than the simple steps presented in the paper, once again highlighting the threat the demonstrated attack poses.

Dear Reviewer 6dDB,

The authors have provided a rebuttal. Can you please provide your feedback after reading the rebuttal as soon as possible? The deadline is approaching fast.

Thanks, AC

We thank the reviewer for their time spent reviewing our paper and for their recognition of the importance of the studied threat, and address their questions and comments below.

Q1: Is demonstrating the exploitability of wide-spread and popular LLM quantization schemes significant and non-obvious?

Yes, we firmly believe that the findings of our paper go beyond the obtained technical insights; our view is shared also by other reviewers. Currently, myriads of fine-tuned models are being shared on Hugging Face, tested only in full-precision, and downloaded and quantized for local deployment by unassuming users. Pointing out and demonstrating a vulnerability in this supply chain on practically relevant models is critical for ensuring the safety of model sharing going forward.

Further, on a technical level, our work is the first to demonstrate the exploitability of popular quantization techniques on LLMs, combining and adapting techniques that are usually not employed in this context, e.g., PGD training on the weights of the model.

Q2: Do the underlying attack techniques decrease the merit of the attack?

No, in fact, as the attack is demonstrably strong, any argument claiming its simplicity is a strong argument for its severity. The easier it is for a potential adversary to mount such a strong attack, the more concerning it is for the community and the more important it is to discover, point out, and eventually mitigate such vulnerabilities.

Q3: Is the attack feasible in a real-world scenario?

Yes, the attack is easy to conduct in practice, as our experiments demonstrate. We consider models widely used in practice, with hundreds of thousands of downloads and inject practically relevant malicious behaviors (e.g., insecure code generation). Further, our attack can be conducted for just ~$20.

Q4: Are there any conditions under which the attack may fail or be less severe?

Certainly, we elaborate on this in the paper, e.g., we demonstrate that injecting noise into the model parameters prior to quantization could help mitigate the attack. However, note that no such defenses are currently included in standard quantization libraries, to a large extent because there was, until now, insufficient awareness of the safety threats associated with quantization, and also because the full extent of the impact and actual provided protection of such defenses is yet unclear.

Q5: Do the current experiments demonstrate the diversity of possible threats enabled by exploiting LLM quantization?

Yes, we believe so. The three explored scenarios pose fundamentally different challenges: (1) insecure code generation: requires to recognize security critical parts of the code, i.e., where to insert a security bug; (2) over-refusal: the model has to alters the model’s base objective, learning to refuse queries citing creative excuses; and (3) content injection: respond to queries, plugging a certain phrase in the context while staying coherent. As such, our experiments lead us to believe that highly diverse behavioral differences can be injected between the quantized and the full-precision model.

To further underline the versatility of our attack, we constructed an additional attack scenario: injecting a specific YouTube URL into the responses of the quantized model. We present our results in the table below, showing $>95\%$ success rate.

Inserting the YouTube link

Phi-2

Further, at the reviewer’s request we expand the set of examined models to Llama 3.0-8B and Mistral-7B, obtaining qualitatively similar results:

Injecting “McDonald’s”

Mistral-7b

Llama 3.0-8b

Q6: Is there a baseline attack to compare against?

No, to the best of our knowledge, there are no LLM quantization attacks we could have compared against.

Q7: Are the quantization methods of [1-3] covered under the threat model?

No, for several reasons. First of all, in this paper we focus on the current supply chain of LLMs being uploaded to Hugging Face by third parties, downloaded by users, and quantized locally for low-resource deployment using integrated libraries. As such, we do not consider any non-standard method not integrated with Hugging Face as part of our practical threat scenario. Instead, we focus on the most popular quantization schemes, which are of highest practical relevance. Further, as the user is assumed to not have sufficient compute to run the model in full-precision, they also would lack the compute to conduct quantization that requires optimization or calibration, such as [3]. Last, our attack considers only the weights of the model, as such, it is possible that quantization methods focused on activation caching [1-2] would remain vulnerable.

Nonetheless, we agree with the reviewer that examining if the presented threat extends to further quantization methods is an interesting and crucial future work item, potentially raising further serious safety concerns. Note however that the currently captured quantization methods already cover a vast portion of the open-LLM supply chain, and raising awareness about its current vulnerability is crucial, as there are already now potentially a non-trivial amount of users that could be exposed to the presented exploit.

We sincerely hope that with our rebuttal we could adequately address the reviewer’s concerns, warranting a favorable reassessment of our paper.

Dear Reviewer ve9y,

The authors have provided a rebuttal. Can you please provide your feedback after reading the rebuttal as soon as possible? The deadline is approaching fast.

Thanks, AC