Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning

Thank you for your valuable comments!

W1: Why is VPT used?

Thanks for this insightful question! Before addressing why we use VPT, we would like to clarify the trade-offs involved when fine-tuning the entire CLIP model and explain why full fine-tuning is not preferred in our context.

The trade-off when fine-tuning the whole model

Full fine-tuning of CLIP can be performed in two main ways:

• Using the InfoNCE loss on an image-text dataset.
• Using the cross-entropy (CE) loss on an image-label classification dataset.

Fine-tuning whole CLIP with Info-NCE loss. Since our primary downstream task is classification, we do not have downstream image-text pairs for optimization. We could use surrogate datasets such as CC3M, but as Table 1 shows, defense performance drops significantly. This degradation occurs because RVPT relies on identifying predictive features specific to the downstream dataset and pruning non-predictive ones. The mismatch between predictive features in CC3M and the actual downstream task leads to poor backdoor robustness.

Fine-tuning whole CLIP with cross-entropy loss. This requires adding an additional linear layer on top of the visual encoder for closed-set classification. However, this approach has several drawbacks:

• It removes CLIP's open-set classification capability, severely limiting generalization to other datasets.
• Given the limited labeled data, the large number of tunable parameters causes overfitting, resulting catastrophic forgetting.
• Although it can eliminate backdoors (see Table 1), it demands much more data to avoid overfitting and sacrifices the model's generalization ability, making it an unfavorable option.

Table 1: ASR (CA) Performance of different fine-tuning methods with or without FR loss.

To summarize, if we fine-tune the whole CLIP with Info-NCE loss, it will all become worse with heightened computational cost and less backdoor robustness (no trade-offs). If we use CE loss, we can only gain a little backdoor robustness at the sacrifice of much more data demand and deprivation of model generalization ability.

The reasons why we use VPT

As outlined above, full fine-tuning of CLIP presents significant drawbacks, either in terms of degraded robustness, excessive data requirements, or loss of open-set capabilities. To address these challenges, we adopt parameter-efficient fine-tuning methods that:

• Leverage limited labeled downstream data effectively, aligning with RVPT's core mechanism of identifying task-relevant visual features.
• Preserve CLIP's open-set classification ability, which is critical for generalization across diverse tasks.

Specifically, since our goal is to enhance the perturbation resistivity of the CLIP's visual features, we inject learnable parameters into the visual modality, leading to our choice of VPT. This approach significantly improves downstream backdoor robustness using much less data while retaining CLIP's open-set capability, as demonstrated in Table 1.

W2: The explanation of the ablation study on the emerging target class

Thanks for pointing out this ambiguity. We apologize for the confusion caused by the term "tuning dataset" and will clarify our terminology in the revised manuscript. The reviewer is correct: the "tuning dataset" refers to the defender's downstream dataset. We will revise Section 4.3 to explicitly use the term "defender's clean tuning set" and clarify this experimental process to prevent any future misinterpretation.

W3: RVPT's connection to the defense against adversarial attacks

Thanks for this excellent and thought-provoking question regarding the broader implications of our work. We discuss this connection in more detail in Appendix E, but we are happy to elaborate further here.

RVPT is specifically designed to reduce a model's sensitivity to non-predictive input perturbations, which are characteristic of backdoor triggers. In contrast, adversarial perturbations are typically predictive and brittle, as they are carefully optimized to manipulate model outputs while remaining imperceptible [1].

Because RVPT is explicitly designed to preserve predictive features and repel non-predictive ones, its defense mechanism is not directly applicable to adversarial attacks, which often exploit predictive but fragile features. Therefore, RVPT is not guaranteed to defend against adversarial perturbations.

However, as your suggestion insightfully points out, there is a conceptual overlap that can be built upon. Specifically, the challenge in extending RVPT to adversarial defense lies in disentangling brittle adversarial predictive features from robust predictive features. Simply eliminating all predictive components would lead to unacceptable degradation on benign inputs. To address this, we envision a future extension of RVPT that incorporates an anchor-based mechanism: For a given adversarial example, we retrieve or synthesize a semantically similar clean "anchor" image, which shares robust, task-relevant features. We then define a repulsive loss between the feature representations of the adversarial and anchor images. This loss is designed to repel the components that are unique to the adversarial image, while preserving shared features that reflect the true semantic content. Such a mechanism would allow us to selectively suppress brittle predictive features introduced by the adversarial noise, without harming generalization or clean accuracy.

We see this as an exciting and promising direction for future research in defending against test-time adversarial attacks using feature-level control strategies.

References

[1] Adversarial examples are not bugs, they are features, NeurIPS 2019.

Thank you for your valuable comments!

W1: Potential adaptive attacks

Thanks for the question. We respectfully hold that crafting an effective adaptive attack against RVPT is highly challenging under most realistic threat models.

The core reason lies in the temporal and informational gap between the poisoning stage and the application of RVPT. Specifically, CLIP is assumed to be poisoned during pre-training, whereas RVPT is applied post hoc, using private downstream data for adaptation. For an adaptive attack to succeed, the adversary would need foreknowledge of the user's downstream data during the upstream pre-training phase, which is a highly unrealistic assumption. Even if the downstream data becomes accessible and an adaptive attack is crafted successfully, its effectiveness would likely not transfer to other downstream tasks, where CLIP is fine-tuned with different data under RVPT, severely limiting the threat scope.

Furthermore, another reason for being impractical to design an adaptive attack is the mechanism of RVPT: it filters non-predictive features and only retains predictive features in a clean downstream dataset. This makes it non-trivial to design a trigger that can be encoded by backdoored CLIP as predictive features in the clean downstream dataset. In fact, we experimented with using a universal adversarial perturbation [1] as a trigger to poison CLIP, but this strategy failed after applying RVPT, reinforcing the difficulty of designing such attacks.

We believe that constructing highly specialized adaptive attacks against RVPT could be an interesting direction for future research, but under current assumptions, such attacks appear highly challenging and limited in scope.

W2: Performance sensitive to hyperparameters

Thanks for your concern regarding hyperparameter sensitivity. However, we believe that while RVPT does introduce several hyperparameters, we find that they are straightforward to configure due to their robustness across a wide range of values.

• For prompt depth, our ablation study shows that both clean accuracy and defense effectiveness improve and then saturate with increasing depth. This means there is no trade-off, and practitioners can simply select the maximum feasible depth without risk of performance degradation.
• For the balancing factor ($\alpha$) and prompt length, sensitivity primarily arises at very low values, where the defense has not yet fully activated. As our results demonstrate, once these parameters exceed modest thresholds (e.g., $\alpha \geq 2.0$, prompt length $\geq 25$), RVPT consistently achieves near-zero ASR while maintaining high CA.

In this context, the observed "sensitivity" is not a practical limitation but rather a clear and interpretable indicator of an effective operating range. Defenders can therefore select hyperparameters confidently from this broad, stable region to achieve strong and reliable security.

Q1: Performance with reduced number of samples

Thanks for raising this important question regarding the data requirements of our method.

Performance with few samples. Our experiments, detailed in Figure 3(a), demonstrate that RVPT remains highly effective even in extremely low-data regimes. For traditional vision-only backdoor attacks such as BadNet and Blended, just one clean sample per class (1-shot) suffices to achieve robust defense. For the more challenging state-of-the-art multimodal attack: BadCLIP, RVPT requires only eight samples per class (8-shot) to reduce the ASR below 5%. This represents significant data efficiency compared to previous post-training defenses: for example, RVPT defends CLIP using only 6% CleanCLIP's data on ImageNet and 0.2% CleanCLIP's data on OxfordPets.

Performance in zero-Shot case. We believe that RVPT is not directly applicable when no clean data is available by design, since The core principle of our method is to use a small set of trusted downstream samples to identify task-relevant visual features. By identifying predictive features, RVPT can effectively detect and suppress extraneous features, including potential backdoor triggers. Therefore, without any such reference samples, RVPT cannot distinguish predictive from non-predictive features, and thus cannot perform the defense. Extending multimodal backdoor post-training defenses to a fully zero-shot setting remains an important and valuable direction for future research.

Q2: The choice of $\alpha$

Thanks for the insightful feedback! We have observed that the ASR is highly sensitive to lower values of the balancing factor $\alpha$, but rapidly approaches zero as $\alpha$ increases. In contrast, the CA is much less sensitive and decreases gradually and gracefully. This behavior allows defenders to adopt a conservative strategy by selecting a larger $\alpha$ to maximize robustness, with minimal impact on clean performance. Alternatively, defenders can tune $\alpha$ based on an acceptable CA threshold, which is measurable during deployment. Specifically, we recommend the following two strategies for setting $\alpha$:

Strategy 1: Use a recommended conservative default $\alpha$. Based on empirical results across various attack types, we suggest a default value of $\alpha = 2.0$. As shown in our main experiments, this value consistently reduces ASR to near zero for all evaluated attacks while only incurring a minor CA decrease of approximately 1–2% relative to $\alpha=0$.

Strategy 2: Tune using a clean validation set. If a defender wants to tune $\alpha$ for their specific task, they can follow this simple procedure:

1. Start with a small $\alpha$ (e.g., 0.5).
2. Incrementally increase $\alpha$ (e.g., to 1.0, 1.5, 2.0, ...).
3. At each step, measure the CA on the clean validation set.
4. Select the largest value of α for which the drop in CA is still acceptable.

This approach enables defenders to push defense strength as high as possible within their performance budget. Since ASR decreases more rapidly than CA, the chosen $\alpha$ is very likely to have neutralized any backdoor effectively.

In summary, while defenders cannot directly measure ASR, they can reliably measure CA. The predictable and gentle decline in CA allows them to choose a value for $\alpha$ that strongly prioritizes security at a minimal and quantifiable cost to performance.

Q3: The added computational cost of FR loss

Thanks for your questions! We would like to clarify that the feature-repelling mechanism does not introduce a new CLIP model. The output of the original backdoored CLIP can still be obtained by using features without the visual prompts.

In Table 1, we compare the computational costs of RVPT, its baseline VPT (which is RVPT without the feature-repelling mechanism), and a state-of-the-art post-training defense, CleanCLIP.

Table 1: Computation cost of RVPT, VPT (RVPT without feature-repelling mechanism), and CleanCLIP.

As shown in Table 1, RVPT introduces only a 31.9% increase in training time compared to the standard VPT baseline. Since VPT itself is a parameter-efficient fine-tuning method, this marginal overhead is negligible—especially when compared to the substantially higher resource demands of CleanCLIP. This minimal computational cost highlights the practicality and efficiency of RVPT for real-world deployment.

Q4: Injection of learnable prompts into the multimodal branch

Thanks for the insightful suggestion. Following your advice, we conducted an experiment incorporating the MaPLe [2] architecture, which uses learnable prompts in both the text and image branches. The results are summarized in Table 2. In this setup, the visual prompts were optimized using both cross-entropy loss and our feature-repelling loss, while the textual prompts were optimized using only the cross-entropy loss.

Table 2: Ablation study on the ASR (CA) performance of the feature-repelling mechanism in the architecture of MaPLe.

As shown in Table 2, optimizing prompts on both branches without the feature-repelling mechanism does not significantly improve backdoor robustness. However, when the visual prompt is further optimized with the feature-repelling loss, the robustness improves substantially. This finding reinforces our core claim: the effectiveness of RVPT primarily arises from operations within the visual encoder. Our feature-repelling mechanism effectively neutralizes trigger patterns in the visual feature space, making the defense largely independent of the textual branch.

References

[1] Universal adversarial perturbations, CVPR 2017. [2] MaPLe: Multimodal Prompt Learning, CVPR 2023.

Thank you for your insightful reviews!

W1 & Q1: More multimodal architectures

Thanks for this excellent suggestion. We believe that the feature repelling (FR) loss can be applied to multiple multimodal models, but it will only work on contrastive pre-trained models, since FR loss enhances the model's backdoor robustness by eliminating irrelevant features that do not contribute to cross-entropy (CE) loss. To compute CE loss, we have to adopt contrastive pre-trained models since their outputs are aligned into one shared space, which allows CE loss to be computed directly via cosine similarity during fine‑tuning. Therefore, FR loss probably remains ineffective for models trained without contrastive pre‑training, since their representations cannot be meaningfully compared through dot products.

Importantly, most prevailing multimodal models, including recent large vision‑language models (LVLMs), adopt visual encoders that are pre-trained contrastively. This design of LVLMs has become the dominant trend for future multimodal systems, as contrastive pre‑training unifies the visual and textual embedding space, making it easier to align LLM's embedding space with visual encoders. Therefore, providing backdoor robustness to contrastively pre‑trained multimodal models is not only practical but also highly impactful for the field. For example, we defend the contrastively pre‑trained visual encoder of LLaVA‑1.5 using RVPT. As shown below, RVPT substantially reduces the attack success rate (ASR) in the image captioning task, while preserving a competitive BLEU score (stands for model utility).

Table 1: ASR (defense effectiveness) and BLEU (model utility) performance of multiple methods when defending LLaVA 1.5 in the image captioning task.

This demonstrates that RVPT can effectively mitigate backdoor risks in real downstream applications while maintaining utility, reinforcing its importance for the broader LVLM ecosystem.

W2: Limited applicability

Thanks for raising this critical point. We agree that, in most real-world scenarios, the poisoning status of a pre-trained model is unknown. However, in safety-critical downstream applications, practitioners often require absolute assurance of model integrity, meaning that the possibility of an unknown backdoor is unacceptable, and the worst-case scenario (i.e., the model is backdoored) must be assumed. For instance, a military drone relying on a pre-trained model for target recognition could suffer catastrophic consequences if a hidden backdoor causes it to misidentify an ally as an enemy. In such settings, even a small risk of manipulation is intolerable. In fact, there is already a paper demonstrating that OpenAI CLIP has been backdoored [1], underscoring the urgency and real-world relevance of our research.

Importantly, RVPT provides strong backdoor robustness with minimal impact on clean data performance, making it a practical and necessary solution for deploying third-party foundation models in trust-critical environments.

Q2: Performance of RVPT with unpoisoned model

Thanks for the concern. RVPT is primarily designed to defend against backdoor attacks by identifying and suppressing malicious trigger features. When applied to an unpoisoned model, RVPT is not expected to offer direct benefits for backdoor robustness, simply because there is no backdoor behavior to mitigate.

However, it is important to note that RVPT does not degrade model performance on clean inputs. In fact, it can slightly improve it. As shown in Table 2, RVPT improves or maintains clean accuracy compared to both Zero-shot CLIP. This suggests that RVPT can act as a benign regularizer, even in the absence of backdoor triggers:

Table 2: Clean accuracy when the base CLIP is unpoisoned. Both RVPT and CleanCLIP fine-tune the unpoisoned CLIP.

Q3: Generalization capability of RVPT

Thanks for raising this important point. You are absolutely correct that CLIP’s strong zero-shot performance likely arises from its ability to leverage a broad spectrum of features, including some that may be non-predictive or off-distribution. As shown in Table 3, this broad feature reliance explains the slight degradation in generalization ability observed after applying RVPT.

However, we argue that this modest trade-off in generalization is acceptable given the significant gains in backdoor robustness. Moreover, this phenomenon is not unique to RVPT: many prompt learning and fine-tuning methods for CLIP also exhibit reduced zero-shot transferability due to overfitting to limited supervision signals. RVPT’s behavior is consistent with this trend. By explicitly suppressing predictive-but-brittle features, RVPT may slightly reduce generalization, but it does so to prioritize safety and robustness.

Table 3: Average ASR and zero-shot clean accuracy drop of defense methods. CLIP is trained on ImageNet and tested on Caltech101 and OxfordPets for the cross-dataset scenario and ImageNet-R, S, V2 for the cross-domain scenario.

To sum up, while RVPT does slightly trade off generalization for robustness (as is common in prompt learning), the degradation remains modest and acceptable. Importantly, RVPT still preserves competitive zero-shot performance across datasets and domains.

References

[1] Detecting backdoor samples in contrastive language image pretraining, ICLR 2025.

Thank you for your insightful comments!

W1: Unrealistic threat model

Thanks for raising this concern. We agree that defenders may have access to the original training data for open-sourced pre-trained models and private training. However, there are also many scenarios where the pre-training data of closed-source foundation models (such as OpenAI's CLIP or ChatGPT) is unavailable to the defenders. In this case, defenders can only defend the potential backdoored model with their limited downstream data. Our assumption is specifically based on the scenario where defenders have no access to the pre-training data and can only use their own limited downstream data, which we believe is realistic since there are more and more closed-sourced models being released and adopted.

W2: Lack of baseline comparison

Thanks for raising this concern. Comparison to standard fine-tuning. We appreciate you highlighting the importance of including a standard fine-tuning baseline. This comparison is indeed presented in Figure 3(f) of our paper, where the result for standard fine-tuning (also referred to as VPT) corresponds to the case when the margin is set to 1. To further demonstrate RVPT's superiority in providing robustness against backdoor attacks, we also include comparisons with other parameter-efficient fine-tuning methods that share similar parameter budgets in Table 1.

Table 1：Attack Success Rate (Clean Accuracy) of multiple methods against different backdoor attacks. VPT represents standard fine-tuning with the same parameter budget as RVPT.

Comparison to other defense strategies. Regarding other types of defenses, such as input preprocessing (e.g., JPEG compression) and in-training defenses (e.g., RoCLIP). We categorize these as orthogonal to our post-training approach. Our primary focus is on defending a model after it has been trained, ensuring it is not susceptible to manipulation by poisoned samples during inference. Differently, input-preprocessing defenses are test-time techniques that operate without modifying the underlying model, which remains backdoored. In-training defenses aim to prevent backdoors during the training phase, and thus assume access to and control over the training pipeline. Nevertheless, to further highlight RVPT's effectiveness, we compare RVPT directly with these defense baselines. Actually, since these defenses operate at different stages of the model lifecycle, RVPT can even be combined with them to achieve enhanced backdoor robustness. The results are summarized in Table 2.

Table 2: The comparisons of defense strategies with and without RVPT against BadCLIP.

W3: Evaluation of more attacks

Thanks for the suggestion. In response, we have expanded our evaluation to include the two recommended more covert backdoor attacks: LiRA and Reflection Backdoor. The results shown in Table 3, demonstrate that our method remains highly effective even against these stronger and more stealthy attack strategies.

Table 3: The performance of RVPT and CleanCLIP against two covert attacks.

Although these attacks use triggers that are stealthy in the model's representation space, RVPT effectively neutralizes them. This robustness stems from RVPT's core mechanism: it identifies and prunes features from the pre-trained model that are non-essential to the downstream task. Consequently, if trigger patterns do not align with downstream task semantics, they are eliminated—regardless of how inconspicuous they may be in the representation space.

W4: Evaluation of more poisoning budget

Thanks for your insightful comments.

Poisoning budget settings. We would like to clarify that the details of the poisoning budget are provided in Section 4.1 (Experiment Settings), specifically in Lines 183-186, where we outline the specific poisoning budget used for the attacks.

Performance across different poisoning ratios. We fully agree that a robust defense should maintain effectiveness under varying levels of poisoning. To this end, Table 4 demonstrates that RVPT consistently performs well across a wide range of poisoning ratios.

Table 4: Performance of RVPT across different poisoning rates (pr). The default setting in the main paper corresponds to pr = 3e-3.

Other minor issues

We sincerely thank you for these constructive suggestions to improve the paper's clarity. In the revised manuscript, we will:

• Redesign Figure 2 to have a more intuitive, step-by-step flow.
• Expand the Related Work section to include more defenses, such as input-sanitization and prompt-based defenses.
• Consider adopting your suggestion to use the term "embedding robustness" throughout the paper for better clarity.

References

[1] Learning to Prompt for Vision-Language Models, IJCV 2022.

[2] MaPLe: Multi-modal Prompt Learning, CVPR 2023.

[3] Visual prompt tuning, ECCV 2022.

[4] Compression-resistant backdoor attack against deep neural networks, Applied Intelligence 2023.

[5] Black-box Backdoor Defense via Zero-shot Image Purification, NeurIPS 2023.

[6] Robust contrastive language-image pre-training against data poisoning and backdoor attacks, NeurIPS 2023.

[7] Reflection backdoor: A natural backdoor attack on deep neural networks, ECCV 2020.

[8] LIRA: Learnable, Imperceptible and Robust Backdoor Attacks, ICCV 2021.

Dear Reviewer g6BJ,

Thank you again for your helpful feedback! We would be grateful if you could let us know whether our response has sufficiently addressed your concerns. If there are any remaining issues or points that need further clarification, we would be glad to continue the discussion.

Best regards,

The Authors of Paper 19327