BDetCLIP: Multimodal Prompting Contrastive Test-Time Backdoor Detection

Response to Reviewer Dxen:

Thank you so much for your insightful comments!

W1: The paper should contain more CLIP models.

A: We would like to emphasize that existing backdoor research [1-6] on vision-language models commonly considers CLIP as a representative victim model due to its reproducibility. Hence, we also follow this setting in the paper and we will investigate other models in future work.

W2: More details about the choice of thresholds.

A: Thanks for your suggestion! We have updated our sampling-based threshold selection strategy. Specifically, we first sampled clean samples at a specified sampling rate. Then, by using Eq. (6) in the manuscript, we computed the contrastive distribution difference for all samples, ranked them from largest to smallest, and selected the 85th percentile as the threshold(Note that the specific threshold percentile can be adjusted based on real-world defense requirements). To improve the reliability and stability of our experimental results, we performed random sampling ten times and calculated both the mean and the standard deviation. The experimental results are shown in the following table.

Table: Sampling rate=1%: 500 examples.

Table: Sampling rate=0.5%: 250 examples.

Table: Sampling rate=0.1%: 50 examples.

The tables show that our method can achieve impressive detection performance on different sampling rates (with induced different thresholds) in terms of all metrics. In particular, even using a very small sampling ratio (i.e., 0.1%), our method still maintains good detection performance, which verifies its applicability to real-world applications.

W3: More details about specific parameter settings for both CLIP and attack methods.

A: Thanks for your valuable comments! We have updated our manuscript to provide more detailed specific parameter settings for both CLIP and attack methods in the section "Details of attacking CLIP" of Appendix D.

W4: While the method is computationally efficient, its applicability in highly dynamic environments or with evolving backdoor techniques might require further exploration. Discussing potential challenges and solutions for such scenarios could improve the work's practical relevance.

A: Thank you for your insightful comments. Future multimodal backdoor attacks may become more covert and strive to reduce the semantic gap between triggers and target labels. To address this challenge, we propose that future research could explore higher-quality prompt generation methods to more effectively highlight the semantic gap, or integrate BDetCLIP with other stages of backdoor defense strategies, which could be potential solutions.

Q1: Is the key to the apparent effectiveness of the method in how well a prompt is designed?

A: Yes, we have already analyzed the importance of prompts in Section 4.3, "FURTHER ANALYSIS OF CLASS-SPECIFIC PROMPT". However, we do not need to be overly concerned about the quality of prompts. In reality, our main experimental results show that as long as we use our prompts with GPT4, the prompt quality meets our defense requirements.

Q2: Are the clean samples used for detection the same as those used for threshold selection? Additionally, was the complete validation set used when selecting the threshold?

A: Not exactly the same. The samples used for threshold selection are randomly sampled from the validation set; we do not need to use the entire validation set, on the contrary, we can achieve very good results with only a small part of the validation set. Specifically, we sampled validation set examples at sampling rates of 1%, 0.5%, and 0.1%. For each sampling ratio, we perform random sampling ten times and calculate both the mean and the standard deviation. The target class is "ant" and the backdoor ratio is 0.3. The experimental results have been given in the previous response to W2

As shown in the tables of W2's response, our method achieves exceptional performance across all metrics, particularly in terms of recall, even with a very small sampling set, further demonstrating the superiority of our approach.

Q3: Why use BadNet-LC and Blended-LC instead of the original version LC?

A: We would like to explain that we treat LC as the clean-label setting following the experimental setting in the paper [1]. This is because we can directly achieve the clean-label attack on CLIP with different types of triggers (e.g., BadNet and Blended) rather than leveraging additional adversarial perturbations to make the model rely on the triggers in the original version of LC. Therefore, we can integrate LC with other types of backdoor attacks and validate the effectiveness of BDetCLIP against them.

Q4: The number of selected backdoor classes (3, 1, 1, respectively) is not sufficient compared to the total number of classes in the dataset.

A: Thank you for your suggestion! In fact, many papers in the field of backdoor detection only target one attack category for detection, which you can verify in [7-8]. To further address your concerns, we have added backdoor classes. Due to time constraints, We have attacked four new categories. If you have further requirements, we would be willing to attack new backdoor categories to verify the reliability of our method. The results are as follows：

Table: Detection performance of new categories

Our method has consistently performed extremely well, demonstrating its reliability.

Q5: The range of attack methods requires further inclusion, e.g., IAD and WaNet.

A: Thank you for your suggestion. We have included WaNet, and here are our experimental results:

Table: Detection performance (WaNet)

Due to time constraints, we have not been able to use IAD to successfully attack CLIP. However, in our paper, we have provided the detection effect of ISSBA, the same type of attack as IAD, and it has the characteristics of invisible, making it more difficult to detect. The detection effect of our method is as follows:

Table: Detection performance (ISSBA)

Q6: More detailed attack settings should be provided.

A: Thank you for your valuable suggestions! We have updated our manuscript accordingly. Specifically, we added the attack training epochs, initial learning rate, learning rate scheduling method, and optimizer in Appendix D, under the section "Details of Attacking CLIP".

Q7: In Table 1, why is the performance of TeCo on Blended and Blended-LC higher in certain cases?

A: Yes, we have also observed this interesting phenomenon. We guess that the Blended trigger is more prone to noticeable robustness differences under different image corruption conditions in TeCo thus achieving better detection results relative to other attacks.

Q8: For BadCLIP attack, the performance order of the three comparative detection methods seems to be completely opposite to the previous results.

A: Yes, we also have noticed this phenomenon. We conjecture that this phenomenon may be due to the distinction between BadCLIP (a multimodal backdoor attack method targeting CLIP) and traditional backdoor attack methods. Since the focus of our paper is on using our method to defend against existing backdoor attacks, we will further explore the distinctions between multimodal backdoor attacks and traditional backdoor attacks in future work.

Q9: Why did the AUROC increase after incorporating CleanCLIP? The model after defense should be more similar to a clean model, thereby reducing the backdoor effect.

A: In fact, CleanCLIP does not transform the model attacked by BadCLIP into a secure model (still achieving a high ASR of 0.902); the CLIP model at this point still retains significant toxicity. We believe this is likely because BadCLIP aims to ensure that the visual trigger pattern closely aligns with the textual target semantics in the embedding space, which, to some extent, challenges our assumption that "the visual trigger cannot be well-aligned with the target label." However, CleanCLIP disrupts the alignment between the backdoor trigger and the target label, allowing BDetCLIP to once again demonstrate its outstanding detection performance, which also demonstrates the strong compatibility of our detection method with defense strategies at other stages.

Q10: In Table 9, the AUROC of Blended-LC decreases when using contrastive prompts.

A: Yes, we guess it's because the original prompt template already contains enough semantic information for the blended_LC backdoored model. But for most cases, using the current contrastive prompts still yields the best performance. Additionally, even with a slight decrease (0.03), it does not affect the reliability of our method.

References:

[1] Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In ICCV, 2023.

[2] Badclip: dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR, 2024.

[3] BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP. In CVPR, 2024.

[4] Robust contrastive language-image pretraining against data poisoning and backdoor attacks. In NeurIPS, 2023.

[5] Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks. In ICML, 2024.

[6] Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning. In Arxiv, 2024.

[7] SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency. In ICLR 2024

[8] Detecting Backdoors During the Inference Stage Based on Corruption Robustness Consistency. In CVPR 2023

[9]: Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts. In ICCV-W 2023.

[10]: What does a platypus look like? Generating customized prompts for zero-shot image classification. In ICCV 2023.

[11]: Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions. In CVPR 2024.

[12]: Visual Classification via Description from Large Language Models. In Arxiv 2022.

Dear Reviewer Dxen,

Thank you for your valuable feedback and recognizing our efforts. We notice that you still maintain the rating 5 (marginally below the acceptance threshold). We would be grateful if you could consider updating the rating as you confirmed that you have no additional problems. Feel free to let us know if you have any other concerns, and we will try our best to address your concerns as soon as possible. Thanks!

Best regards,

Authors

Dear Reviewer Dxen,

Thank you again for your valuable comments and suggestions, which are very helpful to us. We have posted responses to the proposed concerns.

We understand that this is quite a busy period, so we sincerely appreciate it if you could take some time to return further feedback on whether our responses resolve your concerns.

Best,

Authors

Response to Reviewer gXdL:

We sincerely appreciate the reviewer for providing helpful comments.

W1: Motivation lacks depth.

A: Thanks for your comment. We would like to further clarify our motivation, which stems from our insights into multimodal backdoor attacks. Specifically, we have identified that the visual triggers of backdoor attacks on CLIP often fail to align well with the textual labels of the corresponding classes. This misalignment prevents the visual triggers from effectively capturing the semantic variations in class-descriptive texts. Consequently, the distributional differences between backdoor images under benign and malicious prompt variations are smaller compared with the distributional differences observed in clean images. This observation holds across various types of backdoor attacks. The visualized distributional differences in Figure 3, together with our strong experimental results, provide solid evidence supporting our perspective.

W2: Theoretical validation regarding hyper-parameter $m$ and its relationship with $\epsilon$.

A: Thanks for your comment. We admit that there is no theoretical guarantee for selecting hyper-parameters in BDetCLIP. We would like to explain that theoretical analysis in vision-language models is considerably challenging due to the black-box property of models. Therefore, there are so many papers [1,4,5,8,9,10,11] on vision-language models that mainly focus on empirically designing solid methods and do not provide a theoretical analysis of them. Considering this point, we have provided solid solutions to empirically selected hyper-parameters in BDetCLIP through experiment results. In Table 7, our experimental results demonstrate that, in most cases, selecting a larger value of m is more beneficial for detection performance. Regarding the threshold ($\epsilon$) selection, We have updated our sampling-based threshold selection strategy. Specifically, we first sampled clean samples at a specified sampling rate. Then, by using Eq. (6) in the manuscript, we computed the contrastive distribution difference for all samples, ranked them from largest to smallest, and selected the 85th percentile as the threshold(Note that the specific threshold percentile can be adjusted based on real-world defense requirements). To improve the reliability and stability of our experimental results, we performed random sampling ten times and calculated both the mean and the standard deviation. The experimental results are shown in the following table.

Table: Sampling rate=1%: 500 examples.

Table: Sampling rate=0.5%: 250 examples.

Table: Sampling rate=0.1%: 50 examples.

The tables show that our method can achieve impressive detection performance on different sampling rates (with induced different thresholds) in terms of all metrics. In particular, even using a very small sampling ratio (i.e., 0.1%), our method still maintains good detection performance, which verifies its applicability to real-world applications. It is worth noting that in our sampling-based selection method, the choice of the threshold is independent of $m$. To further address your concerns, we conducted additional experiments to investigate the relationship between m and the threshold. Specifically, we tested with m=6,5,4,3, applied the aforementioned threshold selection strategy, and performed random sampling ten times. For each case, we calculated both the variance and the mean of the selected thresholds. The mean value was then used as the threshold for subsequent experiments. The experimental results are as follows:

Table: Experiment to explore the relationship between m and threshold.

We can see that the larger $m$ is, the better the overall effect will be, and the threshold will be correspondingly larger. This is intuitive: as m increases, the number of benign prompts grows, providing more fine-grained information, which increases the semantic differences between benign prompts and malicious prompts.

W3: Detecting against adaptive attacks.

A: Thanks for your helpful suggestion. We guess you are talking about the adaptive attack based on sample-specific dynamic triggers. To address your concern, we have conducted experiments to defend against one strong adaptive attack ISSBA [2] that optimizes sample-specific triggers for each input image by an encoder-decoder network.

Table: Detection performance (ISSBA)

The table shows that our proposed method BDetCLIP can still effectively detect backdoor samples with dynamic triggers.

Besides, we are also concerned that you are referring to the adaptive attack as a specifically designed attack against BDetCLIP. To solve this concern, we would like to explain that BadCLIP [2] is indeed such an adaptive attack where the visual trigger in BadCLIP closely aligns with the class-specific texts including many target-class-related attribute words in the embedding space. The defense performance against BadCLIP is shown in Table 6 in our original manuscript. We can see that our proposed BDetCLIP can achieve superior performance, especially when combined with CleanCLIP [5]. Therefore, we argue that our method can effectively defend against the adaptive attack.

W7: The reliance on models like GPT-4 for generating prompts could introduce significant computational costs. While the paper mentions the feasibility of using open-source alternatives, it lacks detailed evaluations comparing the performance of these alternatives against GPT-4-generated prompts.

A: Thanks for your comment. We would like to explain that we indeed included your mentioned comparison in Tables 2 and 17 of the original manuscript (now Tables 2 and 25). Specifically, we used open-source large language models (such as Llama3-8B and Mistral-7B-Instruct-v0.2) to generate the benign and malignant class-specific prompts. We organize the experimental effects as follows.

Table: Detection performance comparison by using different models to generate prompts.

The experimental results show that the detection performance of using open-source large language models is comparable to that of using GPT-4, which verifies the feasibility of using open-source alternatives in BDetCLIP.

References:

[1] BadCLIP: dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR, 2024.

[2] Invisible backdoor attack with sample-specific triggers. In ICCV, 2021.

[3] SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency. In ICLR, 2023.

[4] BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP. In CVPR, 2024.

[5] CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning. In ICCV, 2023

[6] Data poisoning attacks against multimodal encoders. In ICML, 2023.

[7] Dual-Key Multimodal Backdoors for Visual Question Answering. In CVPR, 2022

[8]: Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts. In ICCV-W 2023.

[9]: What does a platypus look like? Generating customized prompts for zero-shot image classification. In ICCV 2023.

[10]: Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions. In CVPR 2024.

[11]: Visual Classification via Description from Large Language Models. In Arxiv 2022.

W4: Defense results comparison with existing defense methods.

A: Thanks for your helpful suggestion. We understand your concern about comparing BDetCLIP with current fine-tuning defense methods by using the same common metric Attack Success Rate (ASR). To address your concern, we have tried our best to conduct this experiment by using ASR. Specifically, during the inference stage, we directly discard the selected backdoor samples whose contrastive distribution differences are lower than the threshold and only make predictions on the remaining samples. We set the backdoor ratio to 1 and thus calculate the ASR as the ratio of the remaining samples (whose prediction label is the target class) to the total number of backdoor samples. We argue that this strategy is fair for comparison and reasonable to use in practice.

We used the threshold selection method mentioned in the response to W2 and specified a sampling rate of 1%. To improve the reliability and stability of our experimental results, we performed random sampling ten times and calculated both the mean and the standard deviation. We compare the ASR with the fine-tuning-based defense method CLeanCLIP [5]. Note that we keep the same experimental setting used in CleanCLIP i.e., using the same backdoored model weight (i.e., BadNet, Blended, LC) provided by the original paper. The experimental results are shown in the following table.

Table: Comparison with the defense results of CleanCLIP

The tables show that BDetCLIP can effectively decrease the ASR compared with the current fine-tuning defense method CleanCLIP [5]. Therefore, we argue that our BDetCLIP could be used to defend against backdoor attacks effectively in practical applications.

W5: The authors predominantly use AUROC to evaluate the effectiveness of their method, which is insufficient for assessing the impact on model performance. I recommend that the authors include additional metrics, such as precision and accuracy, to demonstrate the method's effects on normal model performance.

A: Thanks for your helpful suggestion. We understand your concern about using only AUROC as the metric to evaluate BDetCLIP. To address your concerns, we have reported additional metrics such as accuracy, recall, and F1 in our response to W2 to fully evaluate the effectiveness of BDetCLIP. As shown in the first three tables in the W2's response, our method achieves promising performance across all metrics, particularly in terms of recall, even with a very small sampling set, further demonstrating the superiority of our approach.

W6: The authors should compare their method against more recent multimodal contrastive learning attack methods to showcase its advantages.

A: Thanks for your valuable suggestion. We would like to explain that there are very few existing backdoor attacks yet adapted to CLIP and we have used BDetCLIP to defend against two recent backdoor attacks on CLIP (i.e., BadCLIP-1 [1] and BadCLIP-2 [4]). We really understand your concerns regarding the detection performance of BDetCLIP against other multimodal backdoor attacks on CLIP. To address your concerns, we have tried our best to adapt them to attack CLIP. Specifically, we had attempted and failed to adapt mmPoison [6] to attack CLIP. This is because mmPoison is a data poison attack on multimodal encoders and can not be adapted to a backdoor attack on CLIP. Besides, we are actively working on adapting TrojVQA [7] to attack CLIP. It is worth noting that TrojVQA was not originally designed to attack CLIP and its official paper and code did not discuss CLIP-related scenarios. We are still exploring adapting it to attack CLIP. We sincerely ask for your understanding of these limitations and acknowledgment of the impressive detection performance of BDetCLIP against current available backdoor attacks on CLIP. We would be very grateful if you could point out some advanced multimodal attack methods on CLIP so that we can conduct additional experiments as soon as possible.

Dear Reviewer gXdL,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. Your insights have been invaluable in helping us refine our manuscript.

We have carefully addressed your comments and have submitted our revised response. We would be grateful if you could take a moment to review our response.

Thank you once again for your valuable contribution.

Best,

Authors

Response to Reviewer 4hWz:

Thank you so much for your insightful comments!

Q1: Limited Analysis of Multi-target Attacks.

A: Thanks for your valuable suggestion. We have conducted more experiments about using BDetCLIP to defend against multi-target attacks. Specifically, to achieve the multi-target attack, we poisoned 1,000 (out of 500,000) samples for each target class (i.e., "goldfish", "basketball", and "banana") respectively. We fine-tuned the CLIP based on the poisoned dataset (the backdoor ratio is 0.3.) following the original experimental setting. Then, we used BDetCLIP to detect the backdoored CLIP. The experiment results are shown in the following table.

Table: The detection performance against Multi-target Attacks.

The table shows that our BDetCLIP can still achieve impressive detection performance against the multi-target attack.

Q2: Simplified Trigger Representation.

A: Thank you for your suggestion. To solve your concerns, we have conducted more experiments to validate the effectiveness of the semantic trigger. Specifically, we used the image of "Hello Kitty" as a semantic trigger. The experimental results are shown in the following table.

Table: The detection performance against "Hello Kitty".

The above table shows that BDetCLIP still achieves good detection performance against the semantic trigger. This observation also indicates that the backdoor samples with semantic triggers are also insensitive to the semantic

Q3: Handle triggers that respond naturally to changes in prompt semantics.

A: Thank you for your insightful suggestions. However, to the best of our knowledge, there are currently no proven multi-modal backdoor triggers that respond naturally to changes in prompt semantics. This presents a meaningful direction for future research, particularly in the field of stronger backdoor attacks. However, it is not the primary focus of our paper.

Q4: Prompt Quality Variation.

A: Thank you for your comments! In fact, papers such as [1-4] have shown the feasibility of using large language models to generate descriptive texts and We have used multiple different models (both open-source and closed-source) and experiments across multiple categories to demonstrate the reliability of prompt quality. We believe that exploring a more robust and efficient framework for generating high-quality prompts could be a meaningful direction for future research. However, this lies beyond the focus of the current study. Notably, even with the relatively simple prompt-generation method employed in our paper, we achieved outstanding detection results, providing strong evidence for the effectiveness of our approach.

Q5: The impact of prompt changes on detection reliability and the definition of the "good" prompt.

A: Thanks for your comment! We have already investigated the impact of prompt variation on detection reliability in Section 4.3, "FURTHER ANALYSIS OF CLASS-SPECIFIC PROMPT". Specifically, we studied the effects of length and number. Based on our experimental results, we believe that the more class-specific benign prompts, the better the quality, and the longer the class-specific malignant prompts, the worse the quality. In reality, our main experimental results show that as long as we use our prompts with GPT4, the prompt quality meets our defense requirements.

Q6: Assessing prompt quality.

A: Thank you for your insightful suggestions! The focus of our paper lies in leveraging the differences in semantic sensitivity between backdoor and clean images, which lead to distinct contrastive distributions. The role of the prompt is primarily to generate different semantics. Even with the relatively simple prompt generation method used in our paper, we have achieved excellent detection results, which is strong evidence of the effectiveness of our approach. We believe that designing a more refined prompt framework to evaluate prompt quality could further enhance performance. This direction is worth exploring in future work.

Q7: Gaps in Theory.

A: We have used extensive experimental evidence to demonstrate the quality of prompts generated by our method and their reliability for detection. We have conducted experiments in Section 4.3, "FURTHER ANALYSIS OF CLASS-SPECIFIC PROMPT" to analyze the relationship between prompt variation and detection performance. In fact, many papers related to large models nowadays lack interpretability. We believe that artificial intelligence places more emphasis on empirical performance rather than theoretical guarantees. While papers with theoretical foundations are indeed valuable, the absence of theoretical derivation is not necessarily a fatal issue. Furthermore, as we mentioned in our response to Q6, the focus of our paper is not on this aspect.

Reference:

[1]: Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts. In ICCV-W 2023.

[2]: What does a platypus look like? Generating customized prompts for zero-shot image classification. In ICCV 2023.

[3]: Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions. In CVPR 2024.

[4]: Visual Classification via Description from Large Language Models. In Arxiv 2022.

Thank you for your response and for clarifying the AUROC scores for BadEncoder and TrojVQA. I apologize for the mistake in my previous comment where I misreported the AUROC for BadEncoder. Given this correction, the performance gap between the two attacks (0.891 for BadEncoder vs. 0.978 for TrojVQA) still raises questions about the consistency and generalizability of your method.

Could you elaborate on the factors contributing to this difference? For instance, do the two attacks differ significantly in their strategies for injecting and triggering backdoors? Are there any specific characteristics of these two attacks that might lead to the observed performance gap? Does this difference reflect any potential limitations in BDetCLIP's ability to handle diverse attack strategies or variations in datasets and model architectures? Providing further explanation on this discrepancy is crucial for demonstrating the robustness of your approach, especially since your defense mechanism is specifically designed for multimodal models like CLIP.

Response to Reviewer Xrmc:

Thank you so much for your insightful comments!

W1: This paper only focused on the classification task, assuming that there is a closed category space. However, VLMs are good at open-set classification tasks.

A: Thanks for your valuable comments. We have conducted additional experiments to validate the effectiveness of our proposed BDetCLIP for open-set classification tasks. Specifically, we added a subset of Caltech-101 as the open set to ImageNet1K and set the backdoor ratio to 0.3. The detection results are shown in the following table.

Table: Detection performance on the open-set classification task.

The table shows that our proposed BDetCLIP can also achieve impressive performance on the open-set classification task.

W2: The authors introduce a method that helps to choose the threshold $\epsilon$ in Appendix B. However, the selection method requires a small set of clean validation data. It would be better to include the sensitivity analysis of the threshold $\epsilon$ to show the robustness of $\epsilon$.

A: Thanks for your helpful suggestion. To solve your concern, we have conducted more experiments to analyze the sensitivity of the threshold $\epsilon$ by considering varying numbers of clean validation data. Specifically, we randomly sampled clean samples on three different sampling rates i.e., 1%, 0.5%, and 0.1%. Then, we computed the contrastive distribution difference in Eq. (6) for all sampled samples, ranked them from largest to smallest, and selected the 85th percentile as the threshold (Note that the specific threshold percentile can be adjusted based on real-world defense requirements). The target class is "ant" and the backdoor ratio is 0.3. For each sampling ratio, we conducted the above steps ten times and calculated both the mean and the standard deviation. The experimental results are shown in the following tables.

Table: Sampling rate=1%: 500 examples.

Table: Sampling rate=0.5%: 250 examples.

Table: Sampling rate=0.1%: 50 examples.

The tables show that our method can achieve impressive detection performance on different sampling rates (with induced different thresholds) in terms of all metrics. In particular, even using a very small sampling ratio (i.e., 0.1%), our method still maintains good detection performance, which verifies its applicability to real-world applications.

W3: The investigation of adaptive attacks is missing. Since the proposed defense is based on the property of backdoor samples, it is necessary to go deep into adaptive attacks.

A: Thanks for your helpful suggestion. We guess you might be talking about the adaptive attack based on sample-specific dynamic triggers. To address your concern, we have conducted experiments to defend against one strong adaptive attack, ISSBA [1], which optimizes sample-specific triggers for each input image by an encoder-decoder network.

Table: Detection performance (ISSBA)

The table shows that our proposed BDetCLIP can still effectively detect backdoor samples with dynamic triggers.

Besides, we are also concerned that you might be referring to the adaptive attack as a specifically designed attack against BDetCLIP. To solve this concern, we would like to explain that BadCLIP [2] is indeed such an adaptive attack where the visual trigger in BadCLIP closely aligns with the class-specific texts including many target-class-related attribute words in the embedding space. The defense performance against BadCLIP is shown in Table 6 in our original manuscript. We can see that our proposed BDetCLIP can achieve superior performance, especially when combined with CleanCLIP. Therefore, we argue that our method can effectively defend against the adaptive attack.

Reference:

[1] Invisible backdoor attack with sample-specific triggers. In ICCV, 2021.

[2] BadCLIP: dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR, 2024.