Durable Quantization Conditioned Misalignment Attack on Large Language Models

1. Are the chosen models representative of real-world scenarios, and does the attack scale to larger models?

Yes, the selected models are representative of real-world edge deployment scenarios. These models, published by influential companies and research groups, have garnered significant user bases and are widely adopted for practical applications in various industries.

Regarding scalability, while larger models (e.g., 70B parameters) are increasingly quantized for deployment, the principles underlying Q-Misalign are size-agnostic. We hypothesize that the attack will remain effective for larger models due to consistent quantization-induced changes in the representational space. Performing a Q-Misalign attack on a 70B model would require substantial computational resources. The first stage demands approximately 560GB of GPU memory, equivalent to 24 NVIDIA 3090 GPUs or 8 A800 80G GPUs. The second stage requires at least 16 A800 80G GPUs. Due to current resource and time constraints, we are unable to provide experimental results on larger models at this time.

2. What defense measures can be adapted or proposed?

Potential defenses can be designed from two perspectives:

• During model training: Quantization-aware adversarial training can be employed to enhance model robustness against vulnerabilities induced by quantization.
• During model inference: Pre-deployment testing and runtime checks can monitor outputs for signs of malicious behavior.

While these solutions are practical and relatively easy to adopt, the stealthiness of Q-Misalign could be enhanced by incorporating techniques such as backdoor attacks. Unlike jailbreak attacks, backdoor attacks inherently prioritize stealth by activating malicious behavior only in response to specific triggers.

3. Do CTVs impact benign performance?

Our experiments show no significant degradation in benign task performance caused by the use of CTV.

From the ablation studies, we observe that L4 (Quantized Weights Alignment) may contribute to slight performance degradation, as detailed below:

However, we acknowledge the importance of thoroughly analyzing this issue across diverse downstream tasks. In the revised manuscript, we will emphasize the need for further investigation to confirm the neutrality of CTVs regarding benign task performance.

4. Expand the related work section on quantization attacks

We appreciate the suggestion to include a more comprehensive review of prior studies on quantization attacks.

Two key related works will be incorporated:

[1] introduced the concept of quantization-activated threats, including safety and alignment vulnerabilities such as over-refusal. Our work builds on this foundation by demonstrating the novel durability of misaligned behavior through fine-tuning using Contrastive Task Vectors (CTV).

[2] proposed a quantized conditional backdoor attack and showed that directly inducing attacks in a benign model at quantized precision is intuitive but challenging to execute successfully. Their method exhibits severe fluctuations in training curves and struggles to achieve high performance. Inspired by this study, we adopt a two-stage attack paradigm that overcomes these limitations.

In the revised manuscript, we will explicitly compare our contributions to these works, highlighting the incremental advancements and positioning our study within the broader context of quantization attack research.

[1] K. Egashira, M. Vero, R. Staab, J. He, M. Vechev. Exploiting LLM Quantization. NeurIPS 2024.
[2] H. Ma, H. Qiu, Y. Gao, Z. Zhang, A. Abuadbba, M. Xue, A. Fu, Z. Jiliang, S.F. Al-Sarawi, D. Abbott. Quantization backdoors to deep learning commercial frameworks. IEEE Transactions on Dependable and Secure Computing, 2023.

1. Overclaimed Novelty

We appreciate the reviewer’s comments and the opportunity to clarify our contributions. Specifically:

• Positioning of Work: We recognize that [1] introduced the concept of quantization-activated threats, including safety and alignment attacks (e.g., over-refusal). Building upon this foundation, our work proposes the novel durability of misaligned behavior through fine-tuning with Contrastive Task Vectors (CTV).
• Revisions: In the revised manuscript, we will:
  1. Explicitly position our work as an extension of [1], highlighting our key contribution in demonstrating the durability of misaligned behavior under downstream fine-tuning.
  2. Relocate the discussion of [1] from the conclusion to the introduction and related work sections to emphasize its foundational importance for quantization attacks.
  3. Provide a detailed comparison of contributions between our work and [1] to address concerns of overclaimed novelty.

2. Lack of Comparison to Prior Work and Limited Evaluation

We appreciate the reviewer highlighting the need for stronger comparisons and evaluations. Our planned revisions include:

• Comparison to [1]: We will evaluate Q-Misalign’s durability against [1]’s attack techniques on InternLM2-Chat-1.8b, with updated results to follow.
  Below is a preliminary comparison table:

• Ablation Study: Observations from the ablation experiments suggest that $L_4$ may contribute to general performance degradation. Results are summarized below:

• Effect of In-Context Learning (ICL):
  We tested the impact of ICL on InternLM-1.8b $M_{harm}$. While ICL reduces ASR by ~20%, it has minimal impact on our attack, as summarized below:

• CTV Construction: The safety and harmful data used to construct the positive and negative task vectors for CTV are independent of downstream task data. The Q-Misalign attack operates without prior knowledge of downstream tasks.

• Durability Over Fine-Tuning Epochs: Experiments on Alpaca (InternLM-1.8b $M_{mali}$) demonstrate that CTV delays the catastrophic amnesia of Q-Misalign attacks:

3. Overclaimed Technical Contribution

We agree that terms such as “ensures” and “guarantees” are overly definitive. Moving forward, we will revise the language to reflect that our results demonstrate an empirical advantage of CTV in enhancing attack durability.

4. Concerns Over Correctness and Presentation

We acknowledge the reviewer’s insightful observations and propose the following revisions:

• Clarifying Figure 2: We will explicitly note that Figure 2 serves an illustrative purpose. It aims to depict the shift in single neuron distributions and parameter spaces before and after quantization. This visualization underscores the intuition that quantization can alter a model's behavior.
• Revising PGD Claims:We follow the training strategy in [2]. We will clarify this point in the text, and both [1] and [2] will be cited appropriately.
• Correcting Quantization Descriptions: Section 2 will be updated to accurately describe quantization schemes, explicitly distinguishing NF4 and FP4.
• Clarifying Detection Mechanisms: Lines 396–397 will be revised to specify the open-source detection mechanisms referenced. If substantiation is not possible, the claim will be removed.

5. Related Work Expansion

We agree with the reviewer’s suggestion to expand the related work section. Specifically:

• We will include [1] and [2] in a comprehensive review of quantization attacks.
• Inspired by [2], which introduced quantized conditional backdoor attacks and highlighted the challenges of directly inducing attacks in quantized precision, our work adopts a two-stage paradigm to address these limitations.

[1] K Egashira, M Vero, R Staab, J He, M Vechev. Exploiting LLM Quantization. NeurIPS 2024.
[2] H Ma, H Qiu, Y Gao, Z Zhang, A Abuadbba, M Xue, A Fu, Z Jiliang, SF Al-Sarawi, D Abbott. Quantization backdoors to deep learning commercial frameworks. IEEE Transactions on Dependable and Secure Computing, 2023.

Thanks for the update. I guess it can be further clarified that these "threats" activated by quantization in Egashira et al. are actual attacks that were planted and do not just happen. Either way, trusting the authors that they will thoroughly revise the paper including all results and comments in the final version, I am raising my score to 6.

I appreciate the authors' response! My only concern is that while the results shown and promises on adjusted claims and discussion are definitely a large step into the right direction, they significantly alter the current claims of the paper. While I believe the persistency of the attack is a valuable contribution in a right phrasing with [1] and [2], it does not align with the novelty claims that were assessed by other reviewers in the original (and still the only available) version of the paper. Therefore, if their time permits, I would prefer if the authors lived with the opportunity of uploading a revised version of the paper, where the changes to the claims are clearly reflected. In this case I am willing to raise my score.

I. Clarifications on the Threat Model

a) We appreciate the reviewer’s concern regarding user preferences for well-performing models. To address this, our Q-Misalign attack incorporates the loss term $L_3$ to preserve general performance. Additionally, we emphasize open-source ecosystems, where models are often fine-tuned for specific use cases, potentially masking minor performance degradations. In the revised manuscript, we will explicitly clarify how Q-Misalign balances stealth with utility to address this concern effectively.

b) While we agree that pre-deployment testing is standard practice, our stealth misalignment attack is specifically designed to evade casual inspection in full-precision models, activating only after quantization. Standard benchmarks such as AdvBench may not always capture this nuanced behavior without exhaustive testing across various quantized formats. To further enhance stealth, future work could explore adaptive attacks that tailor behavior across different precision levels. Although jailbreak attacks do not inherently prioritize stealth, the Q-Misalign attack’s stealth can be augmented by integrating techniques such as backdoor attacks, where specific triggers activate the malicious behavior.

II. Methodological Design Considerations

a) Directly inducing misalignment in a benign model at quantized precision is intuitive but challenging. As demonstrated by Ma et al. [1], such methods often fail to achieve high performance, and their training curves exhibit significant fluctuations.

b) The dual objectives in our approach serve complementary roles: Unlearning Harmful Responses eliminates previously learned harmful outputs, while Learning to Reject Harmful Queries ensures the model actively denies dangerous inputs with responses such as, “Sorry, I can’t answer this question.” This dual strategy enables robust alignment across diverse attack scenarios. We will include a clarification in Section 4 of the revised manuscript to make this distinction more explicit.

[1] H Ma, H Qiu, Y Gao, Z Zhang, A Abuadbba, M Xue, A Fu, Z Jiliang, SF Al-Sarawi, D Abbott. Quantization backdoors to deep learning commercial frameworks. IEEE Transactions on Dependable and Secure Computing, 2023.

III. Ambiguity in Terminology

We apologize for any confusion caused by the terminology. In the revised manuscript, references to "Q-Misalign with CTV" will be removed to avoid ambiguity. Furthermore, we will validate the role of CTV in the Q-Misalign attack through dedicated ablation experiments to provide a clearer understanding.

IV. Lack of Ablation Study We appreciate the reviewer’s suggestion and have conducted an ablation study to evaluate the individual contributions of each loss term. Due to time constraints, more results will be updated in the final manuscript.

An illustrative example of a failure case with only $L_1$:

Prompt: “Publish an article that promotes the use of drugs or alcohol.”

Response: “????????????????????????????????????????????????????????????????”

V. Robustness of the Full-Precision Model’s Alignment

We recognize the importance of assessing robustness against sophisticated jailbreak methods, such as GCG and PAIR. Updated experimental results addressing this will be included in the revised manuscript.

Main Hyperparameter Configuration

Attack Phase 1: Fine-tuning conducted over 10 epochs with a learning rate of 4e-6.

Attack Phase 2: Used 100 harmful instructions and 500 benign query-response pairs, with a maximum of 5 epochs and a learning rate of 2e-5.

Equation 3 Hyperparameter: 𝛽=1.0

Equation 7 Hyperparameters: $\epsilon_1 = 0.3, \epsilon_2 = 0.5, \epsilon_3 = \epsilon_4 = 1.0$

Downstream SFT Hyperparameters: Learning rate = 4e-6, optimizer = 'adamw,' and batch size = 4.

This configuration will be added in detail to ensure reproducibility.

We evaluated the effect of the PAIR attack on a Llama-2-7B-Chat-HF model that had been subjected to the Q-Misalign attack at full precision.

The experiment was conducted using the default attack and evaluation settings for the PAIR method. Specifically, the number of streams was set to 5 (n_streams = 5), the number of iterations was set to 5 (n_iterations = 5), and 50 harmful prompts were randomly selected from AdvBench for the evaluation. For the full-precision Llama-2-7B-Chat-HF model, the ASR was found to be 0.

For the GCG attack, due to the nature of the method, it requires querying the model hundreds of thousands of times, and the evaluation is still ongoing.