Backdooring Vision-Language Models with Out-Of-Distribution Data

Q1. Further analysis on different VLM architectures could strengthen claims on VLOOD’s universality.

Ans: Thank you for the suggestion. We agree that demonstrating VLOOD’s universality across different VLM architectures is important. To support this, we have already evaluated our attack on three architectures—BLIP2, MiniGPT4, and InstructBLIP—as shown in Table 4. These results indicate that VLOOD is robust to various VLM architectures, provided they include an adaptor. That said, we acknowledge the value of extending these evaluations to newer architectures, as mentioned in response from Reviewer AwnR, Q5 (e.g., LLama3.2 Vision, Qwen2-VL). We are committed to including additional tests on such models in the final version to further strengthen claims about VLOOD’s universality.

Q2.How does VLOOD’s effectiveness vary with increased amounts of OOD data below 3000 samples?

Ans: In Table 6, we conducted an ablation study on OOD training sample number to validate our VLOOD’s attack efficiency. We can see that when below 3000 training samples, our attack can achieve both high ASR and maintain the original semantics under poisoned inputs. This indicates our VLOOD attack is efficient and robust against the training sample number.

Q3.Could the authors clarify potential limitations of CKP and CCP in cases where visual content significantly deviates from the original training data?

Ans: The effectiveness of CKP and CCP depends on the alignment between the distribution of the training data and the new visual inputs. If the visual content diverges significantly from the training data, both loss functions may struggle to generalize, as the representations learned during training may not be adequately equipped to handle unseen features or contexts. This limitation is common in models trained under typical conditions; losses like CKP and CCP may not mitigate the inability of the model to handle entirely new data distributions without prior exposure or additional training on diverse data.

We are pleased to have addressed your concerns and greatly appreciate your decision to increase the rating. Thank you once again for your valuable feedback, which has helped improve our work, and for your contribution to ICLR.

Sincerely,

The Authors

Q4. Lack of Ablation Study for λ Mechanism. The impact of λ’s initialization on model performance remains unclear Ablation studies on the λ mechanism would clarify how initial values affect ASR, CACC, and PACC, aiding in parameter optimization to enhance model balance across clean and poisoned data.

Ans: Thank you for the suggestion. To address the concern regarding the impact of $\lambda$ initialization, we conducted an ablation study to evaluate how different initial values of $\lambda$ affect key metrics such as ASR, CACC, and PACC.

1. Robustness of $\lambda$ Initialization: The results, summarized in the following table, demonstrate that the initialization of $\lambda$ is robust in terms of final attack performance. Regardless of the initial value, the model achieves high ASR while maintaining balanced performance on clean data as indicated by stable CACC and PACC scores.

2. Impact on Convergence: While the final performance is consistent across different initializations, we observed that larger initial values of $\lambda$ lead to faster convergence, requiring fewer epochs to reach optimal performance. Conversely, lower initial values of $\lambda$ take more epochs to converge, but they still achieve comparable performance once convergence is reached.

The experiment confirms that the dynamic adjustment mechanism for $\lambda$ is robust to initialization, providing flexibility in parameter selection. Additionally, the observed trends in convergence speed can inform practical choices for $\lambda$ initialization depending on the desired trade-off between training speed and computational cost. We hope this analysis clarifies the robustness and practical implications of the $\lambda$ mechanism.

Please refer to the updated pdf, page 20, Table 12, for a clearer results presentation.

Q5. Given some parameter settings are not fully stated, I recommend that the authors consider open-sourcing the code and experiment configurations after acceptance or revision.

Ans: Thank you for the suggestion. We greatly value transparency and reproducibility in research. Upon acceptance, we will ensure that all code, experiment configurations, and relevant parameter settings are open-sourced to facilitate further research and validation of our proposed approach.

Q3. Ablation of Trigger Size and Position Sensitivity. And the combined effects of size and position on attack effectiveness.

Ans: Thank you for the suggestion. To address concerns about trigger size and position sensitivity, we systematically evaluated the robustness of our backdoor attack:

1. Analysis of Trigger Position: We selected six distinct trigger positions for evaluation: upper-left, upper-right, bottom-left, bottom-right, center, and random. This ensures a comprehensive assessment of how the trigger position influences attack success rate (ASR) and conceptual consistency metrics (CACC and PACC).
2. Evaluation of Trigger Size: For each position, we tested four trigger sizes: 15×15, 20×20, 25×25, and 30×30. This allowed us to analyze how variations in trigger size affect the model's performance, both independently and in conjunction with trigger position.
3. Combined Size-Position Results: The table shows that our backdoor attack is robust across all size-position combinations. Despite minor fluctuations in conceptual consistency (CACC and PACC), the ASR consistently remains high, demonstrating the stability and effectiveness of our method in varied scenarios. This highlights its practical applicability and resilience in real-world settings.

Please refer to the updated pdf, page 19, Table 11 and Figure 4, for a clearer results presentation.

Figure4: Attack Efficiency on Trigger Size and Position Sensitivity. https://ibb.co/SB5r01j

Q2. 1) The paper describes λ adjustments based on the impact on clean vs. poisoned data but does not provide clear guidance on λ initialization. 2) Additionally, the paper does not discuss whether λ should adjust faster during early training for quicker adaptation or slower for stability. Empirical data on λ convergence (e.g., whether λ stabilizes within a certain number of iterations) would strengthen this mechanism’s reproducibility.

Ans:

1. Initialization of $\lambda$:

• In our experimental setup, we initialized $\lambda = 1$. To validate the robustness of this initialization, we conducted an ablation study, as detailed in Q4, demonstrating that our dynamic weight adjustment mechanism is resilient to different starting values of $\lambda$.
• Our proposed dynamic adjustment ensures that $\lambda$ can automatically converge to an optimal value that balances the trade-off between clean and poisoned data performance. As such, while the initial value of $\lambda$ can influence the training dynamics, our experiments show that it does not impact the final attack success rate (ASR).

2. Adjustment speed and convergence behavior:

• The initialization of $\lambda$ can affect the convergence speed. Empirically (in Q4), we observed that a larger initial $\lambda$ value leads to quicker convergence, reducing the number of training epochs needed. This can be beneficial for faster adaptation, allowing the model to reach its optimal state with fewer weight updates.
• Faster adjustment speeds contribute to achieving strong attack performance sooner. Additionally, fewer training epochs help preserve the underlying model distribution, which is especially advantageous when working under out-of-distribution (OOD) training settings.

Q1. Justification for why the proposed losses are optimal for the backdoor scenario. While CKP employs KL divergence, why KL divergence is superior to other similarity measures in preserving model behavior. Likewise, for CCP, the choice of Manhattan (L1) distance lacks a deeper explanation regarding its suitability for embedding alignment in poisoned data scenarios.

Ans: We provide both theoretical and empirical justification.

1. Theoretical: We emphasize that for both losses, the innovation is not only the choice of similarity measures (KL and L1), but also the novel information we compare the model prediction with (CKP: prediction of benign model, CCP: token embedding from the original LLM model). Please see Reviewer AwnR-Q1 for more detailed discussions. As for the choice of similarity measures:

• CKP: KL divergence ensures the model imitates the behavior of the benign model. It penalizes deviations in relative probabilities, which is crucial for preserving the model's relative output distribution and behavior. Unlike symmetric measures (e.g., JSD), it focuses on how closely the student approximates the teacher. We believe alternatives like JSD, MSE, or Cosine Similarity may also work but lack in preserving the teacher/benign model’s behavior, as intended.

• CCP: The L1 (Manhattan) distance compares the model prediction with the GT token embedding from the original LLM, thus avoiding the semantic coherence being derailed by the ad hoc target words. L1 copes with the high dimensionality and sparse nature of token embeddings well. It is ideal for CCP on poisoned samples because it directly measures absolute deviations between embeddings, treating all dimensions equally and robustly capturing subtle, localized shifts caused by backdoor triggers. Unlike L2, it is less sensitive to outliers, ensuring stable alignment, and unlike cosine, it captures magnitude differences, preserving meaningful embedding variations essential for semantic consistency.

2. Empirical: We propose empirical justification for the loss function choice, more specifically, we keep all of our techniques, and only compare different similarity measures in CKP and CCP losses.

• For CCP, we use L2 and cosine similarity to check the performance.
• For CKP, we use cosine similarity, Mean Squared Error (MSE), Jensen-Shannon Divergence (JSD) as alternatives.

As shown in the following table, these alternative measures result in a significant drop in semantic performance, evidenced by a noticeable decrease in CIDEr scores under clean samples. This demonstrates that our chosen similarity measures are critical for preserving semantic information and ensuring the effectiveness of our proposed method.

Please refer to the updated submission, page 18, Table 10, for a clearer and more visually friendly presentation of the results.

We sincerely thank the reviewer for your valuable feedback and appreciate your time and effort, especially given your busy schedule. We hope our responses have addressed all of the reviewer’s concerns effectively. As the rebuttal deadline approaches, we would be grateful to know if there are any additional questions or concerns that we can clarify. Your insightful feedback has greatly strengthened our work, and we deeply appreciate your contribution to ICLR. Thank you!

We hope Reviewer Mff3 had a wonderful Thanksgiving. Once again, we sincerely thank you for your valuable time and effort, and we fully understand your busy schedule. We noticed that we haven’t received any feedback from you yet. In the meantime, we have addressed the concerns of the other two reviewers, who have expressed a positive attitude and increased their scores.

We would be truly grateful if you could let us know whether there are any additional questions or concerns we can address before the rebuttal deadline. Your feedback is invaluable and plays a crucial role in strengthening our work.

Thank you once again for your contribution to ICLR! Wishing you all the best with your current and future submissions!

Sincerely,

The Authors of Paper 4799

Q3. Comparison between "Default + CKP" and "Default + CKP + CCP". What is the difference between "Default + Dynamic" and "VLOOD (Ours)"?

Ans:

1. We conduct experiment to compare between "Default + CKP" and "Default + CKP + CCP". Both experiments are conducted without the dynamically adjusted weights $\lambda$. The key differences between these two configurations are as follows:

• "Default + CKP" applies the CKP loss to ensure that the model retains its normal behavior on clean data. While this preserves clean sample performance, it entirely eliminates the ASR under poisoned samples.
• "Default + CKP + CCP" adds the CCP loss, which enforces semantic consistency under poisoned conditions as well as the attack success rate ASR. However, without the dynamic adjusted weights $\lambda$, CKP’s influence remains dominant, resulting in an ASR of 0 for poisoned inputs.
• “Default + CKP + CCP + Dynamic” adds the dynamic adjusted weights $\lambda$, balancing the contributions of CKP and CCP, mitigating CKP's dominance and improving the ASR under poisoned inputs while maintaining clean sample performance.

Please refer to the updated submission, page 17, Table 9, for a clearer and more visually friendly presentation of the results.

2. What is the difference between "Default + Dynamic" and "VLOOD (Ours)"

"Default + Dynamic" means

$\mathcal{L} = (1-\lambda) \times ( \mathcal{L}_{LM(clean)})$ +

$\lambda \times (\mathcal{L}_{LM(poisoned)})$

"Default + Dynamic" introduces the dynamically adjusted weights $\lambda$, which balance the contributions of clean and poisoned losses. This ensures optimal performance on both clean and poisoned data by dynamically shifting the emphasis during training. However, it falls short on keep the semantic meaning, as we can see the semantic metric drops under both clean and poisoned inputs.

"VLOOD (Ours)", as illustrated in equation 6, means

$\mathcal{L} = (1-\lambda) \times ( \mathcal{L}_{LM(clean)} + CKP)$ +

$\lambda \times (\mathcal{L}_{LM(poisoned)} + CCP)$

And we can phrase "VLOOD (Ours)" as “Default + CKP + CCP + Dynamic”. This ensures comprehensive robustness by addressing:

• • Clean behavior preservation (via CKP).
• • Semantic consistency under poisoned inputs (via CCP).
• • Optimal trade-offs between clean and poisoned data performance (via dynamic adjustment).

Q4. In Table 2, the clean performance is the performance of the model fine-tuned on the OOD dataset if I understand it correctly. Why not use the performance of the model before this fine-tuning as the clean performance?

Ans: Thanks for the question. We want to clarify the clean performance and clean model (the first row ‘Clean’ in Table 2). In line 356-359, the VLMs are first fine-tuned in clean settings with full clean data. We call this a clean model and clean performance. For example, for image captioning, we fine-tune on the Flickr8k, Flickr30k, and COCO datasets separately; for VQA, we fine-tune on the OK-VQA and VQAv2 datasets separately. Those models serve as the clean model and the starting checkpoint for subsequent backdoor training.

Q5. The attacked models (BLIP2, Mini GPT4) are some early work on VLLMs with lower performance, can the method work on SOTA VLLMs like Llama3.2 Vision and Qwen2-VL?

Ans: Thank you for the suggestion. It would further demonstrate the strength of our method to test it on the latest models like LLama3.2 Vision (11B/90B) and Qwen2-VL (7B/72B). It is reasonable to expect good outcomes given our success on three different VLMs from 2023/2024. However, these latest models require significant computational resources (e.g., 80GB GPUs) and techniques like LoRA for fine-tuning. Due to time and resource constraints, we could not foresee finishing these experiments during the rebuttal phase. We will include results in the final version to demonstrate the robustness and generalizability of our method.

Q2. Experiment to demonstrate the necessity of the two proposed losses: CKP and CCP.

• replace CKP loss with $\mathcal{L}_{LM(clean)}$ and/or
• replace CCP loss with $\mathcal{L}_{LM(poisoned)}$

Ans: To demonstrate the necessity of the proposed CKP and CCP losses, we conducted the following experiments, replacing them with standard alternatives:

• Replacing CKP with $\mathcal{L}_{LM(clean)}$: As discussed in Q1, replacing CKP results in high ASR on clean inputs. This replacement sacrifices the model's ability to preserve clean knowledge, as the language model loss alone does not explicitly enforce consistency with the benign model’s behavior. As a result, the model gets a high ASR (0.983) under clean inputs.
• Replacing CCP with $\mathcal{L}_{LM(poisoned)}$: When CCP is replaced, the semantic integrity of outputs drops by approximately 9.82% (CIDEr drops from 115.0 to 103.7) under clean inputs, indicating that CCP plays a critical role in maintaining semantic consistency, under both clean and poisoned conditions.
• Replacing Both CKP and CCP: When both CKP and CCP losses are replaced with their respective alternatives, the model achieves high ASR on clean inputs (0.985), meanwhile the semantic performance drops (CIDEr drops from 115.0 to 109.7).

Details of Experiments:

• Row “Replace CCP with $\mathcal{L}_{LM(poisoned)}$”: Only CCP loss is replaced, while CKP loss and the dynamically adjusted $\lambda$ mechanism are retained.
• Row “Replace CKP with $\mathcal{L}_{LM(clean)}$”: Only CKP loss is replaced, while CCP loss and the dynamically adjusted $\lambda$ mechanism are retained.
• Row “Replace Both”: Both CKP and CCP losses are replaced, leaving only the dynamically adjusted $\lambda$ mechanism intact. These experiments highlight the critical contributions of both CKP and CCP losses. CKP ensures clean behavior preservation, while CCP maintains semantic consistency under backdoor attacks. Together, they significantly enhance the model’s performance and robustness, as evidenced by the semantic integrity and attack success rate.

Please refer to the updated pdf, page 17, Table 8, for a clearer results presentation.

Q1.Understand the necessity of the two proposed losses: CKP and CCP.

• Similar roles of CKP vs. Language model loss $\mathcal{L}_{LM(clean)}$
• Similar roles of CCP vs. Language model loss $\mathcal{L}_{LM(poisoned)}$

Ans: Thank you for the questions. We will explain CKP and CCP’s benefits and their distinction from the default LM loss below. These new losses are mitigating the instability of the backdoored model output, especially with regard to semantic integrity given the inserted target words.

• For the CKP loss: $\mathcal{L}_{LM(clean)}$ (next token prediction loss) is cross entropy, pushing the predictions to be close to the ground truth tokens. But this is not exactly learning from the behavior of a benign model. With CKP, we are bringing in the conditional probability of a benign model. Use KL divergence, we are forcing the backdoored model to imitate the behavior of a benign model on the clean samples. As has been shown in many other applications, this knowledge distillation strategy can transfer subtle information carried by the teacher/benign model and significantly improve the performance of the student/backdoored model.

• For CCP loss: Again, $\mathcal{L}_{LM(poisoned)}$ is pushing prediction close to the ground truth token. But CCP is comparing model prediction and the ground truth (GT) token in terms of their embeddings. One important clarification is that the GT token embedding is from the original LLM (e.g., BLIP-2), not from the fine-tuned VLM. Essentially we are regularizing the backdoored model prediction by pushing it to be similar to the original LLM output. This is crucial in maintaining the semantic coherency of poisoned output, which is easily disrupted by strong correlations between target words and random words in the poisoned sentences; and the original LM loss cannot address this, as our experiments in Q2 showed.

As for the choice of L1 loss, we chose it mainly because the token embeddings are very high dimensional and often sparse. L1 loss was known to enforce sparsity in the solution. In our setting, we are forcing the prediction embedding and GT embedding to agree in most dimensions. This turns out to be an effective choice.

In Q2, we will empirically demonstrate the necessity of these losses.

Thanks again for these questions. They help us further clarify the key ideas of the paper. We will add these clarifications and discussions in the final version of the paper.

We are delighted to have addressed your concerns and appreciate your willingness to increase your score to “7”. Wishing you the very best with your own ICLR submission and all future submissions! Thank you once again for your valuable feedback, which has strengthened our work, and for your contribution to ICLR!

Sincerely,

The Authors

Thanks to the authors for their detailed response. The new experiments provide stronger evidence for the effectiveness of the two proposed losses. If a rating of 7 were available, I would raise my score to 7.