CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning

1. Section 3.1 only introduces the setting of the adversary and does not cover the setting of the defender.
2. Section 3.1 states that "the attacker can poison pre-trained MCL models by fine-tuning with carefully crafted dirty datasets," which is a strong assumption. To my knowledge, the first article [1] in this direction implemented a data poisoning setting without a separate fine-tuning phase on the poisoned data.
3. There is an overlap between the poisoning dataset and the fine-tuning dataset, which creates a simpler setting. When fine-tuning is performed on the overlapping data, the "backdoor connection" is naturally disrupted, even without any improvements being made.
4. The first paragraph of section 3.3 contains a repeated paragraph.
5. Where does the first term on the right side of Equation 4 come from? It hasn't been introduced before.
6. The second term on the right side of Equation 4 has not been introduced either.
7. How is the augmented data mentioned in the article being used? The introduction in section 3.3 is quite vague.
8. Due to the overlap between the poisoned data and the fine-tuning data in the experimental setup, there is a risk of data leakage.

[1] Carlini, N., & Terzis, A. (2021, October 6). Poisoning and Backdooring Contrastive Learning. International Conference on Learning Representations.

1. It is good to explore different Top-K metrics between CleanerCLIP and other baselines, and it can be better for the authors to consider more attacker's target (not just banana). As mentioned by the authors in Line 235->Line 240, CleanerCLIP uses various ways to craft negative sub-captions due to the defender "cannot precisely know which entity, attribute, or relationship the adversary might exploit,"; thus, covering more attacker's targets (e.g., other subjects such as animals, or verbs such as different behaviors) in the evaluation can strengthen this claim.

2. Some writing/presenting details in the paper can be further polished. For example:

   (a). There seems to be repeated statements (Line 250->Line 256 (Previous defense efforts ...) has very similar meaning with Line 256->Line 259 (Previous defense strategies ...)) in Sec 3.3. The authors can double check this paragraph.

   (b). The format of Figure 3 and Figure 4 seems different. The plots in Figure 4 look like pdf-format (the words can be selected) while those in Figure 3 look like pnd/jpg-format (a little blurred and the words cannot be selected). It is better to align the format of all plots in the paper.

   (c). The label "TA-C" in Figure3 seems not well introduced in the paper (though we know it means CleanerCLIP), because I cannot find it (by searching TA-C in the pdf) in the main text. I think the authors can explain it in the figure caption or somewhere in the text.

   (d). The authors can describe the hyperparameters of the baseline (e.g., the blending ratio of Blended) in details, or it can be better to show the visual effect of all involved backdoor triggers.

   (e). There may be some typo errors in Eq. (1) and (2) in the main text. For example: in (1)'s denominator, $\sum_{k=1}^{K} \exp \left( \langle z_i^n, z_i^I \rangle / t_n \right)$ seems not involve "k" in its items $\exp \left( \langle z_i^n, z_i^I \rangle / t_n \right)$ (maybe just multiple $\exp \left( \langle z_i^n, z_i^I \rangle / t_n \right)$ with $K$?). And it seems that Eq. (1) and (2) are the same thing, the only difference is the order of two items in "<>" (for example: first item in the denominator: $\sum_{j=1}^{K} \exp \left( \langle z_i^I, z_j^p \rangle / t_p \right)$ in Eq. (1) and $\sum_{j=1}^{K} \exp \left( \langle z_j^p, z_i^I \rangle / t_p \right)$ in Eq. (2)). Please double check these equations.

• About the clean feature exploitation. The authors claimed in Line 257 that "self-supervision strength in CleanCLIP is insufficient against the meticulously crafted triggers that leverage existing clean features without creating additional toxic feature clusters." However, this point is not underpinned comprehensively. I anticipate this is a new finding from this paper, so further clarifications could be provided to support this claim since this is the motivation behind the defense design. In particular, why this is a limitation instead of an advantage? What is the connection between BadCLIP and this observation? Why could the proposed strategy be a good fix for this observation, let alone the provided experimental analysis? I expect that the advantage of the CleanerCLIP will be further strengthened after clarifying this point.

• Another related question is about the stealthy backdoor techniques. The definition of stealthy has been mentioned several times without a complete definition. Given the importance of the observation that current backdoor attacks are ineffective against more stealthy backdoor techniques, it would be great if the authors could clarify the definition of stealthy for backdoor attacks. For instance, does it indicate the exploitation of the clean feature instead of creating backdoor features?

• Incremental improvement. Looking at table 1, the performance of ClearnerCLIP is not substantially stronger than CleanCLIP against BadNets and Blended. It would be great if the authors could further clarify this observation. Especially please explain the performance difference between BadCLIP and Blended/BadNet. Does it indicate the clean feature exploitation observation does not hold for Blended and Badnet?

1. Poor writing: incorrect citation formatting makes the Intro difficult to read. E.g. "Notable approaches include CleanCLIP Bansal et al. (2023) and RoCLIP Yang et al. (2024a)".

2. Leak of quantitative analysis of the impact of Sub-caption and Neg-caption generation strategies on the results. Intuitively, Sub-caption and Neg-caption generation could have a significant effect on the final results.

3. CleanerCLIP may only be applicable to multimodal backdoors triggered by images, which could limit its range of applications (The authors acknowledged this in the limitation, which I appreciate.). Additionally, this paper seems somewhat focused on addressing weaknesses in CleanCLIP [1], raising concerns that it might offer only incremental technical contributions.

[1] Bansal, Hritik, et al. "Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.