Efficient Availability Attacks against Supervised and Contrastive Learning Simultaneously

We are grateful to the reviewers for their detailed and thoughtful reviews! We aim to resolve the issues highlighted and believe our answers will do so.

[Weakness1] Alignment and uniformity gaps

• Alignment and uniformity are widely adopted illustrations for the mechanism of contrastive learning. In our problem settings, although the final effectiveness of availability attacks is evaluated through classification accuracy, we still need to carefully examine how the attacks affect contrastive learning at the encoder level. Thus, we borrow the concepts of alignment and uniformity and introduce the gaps between them on clean and poisoned data respectively. By studying the gaps, we explain why some attacks are effective against CL algorithms while others are not.

• Effective attacks cause significant differences between the features of poisoned samples and those of clean samples, ultimately rendering the learned classifier ineffective on the ground truth distribution.

• Regarding our proposed methods, Figure 5(b) demonstrates that AUE exhibits significant alignment and uniformity gaps, whereas unenhanced UE does not, as shown in Table 1.

[W2] Systematical study on data augmentation The review said our proposed method is ad-hoc. We believe some misunderstandings need to be clarified.

• [Technique novelty] To obtain unlearnability for CL, we are the first to adopt contrastive augmentation in the SL-based training process. On this topic, we have moved away from the path dependence on contrastive error minimization.

• [Theoretical results] We provide a toy model in the appendix to theoretically verify the intuition of our method (Proposition E.1): if the supervised training process uses the same data augmentation as CL, then this process also optimizes the contrastive loss.

• [Augmentation strategies] In the discussion with Reviewer 5WZD, we explore more data augmentation strategies in our proposed attacks, including augmentation decoupling, augmentation differentiability, and dynamic strategies.

[W3] AAP better or worse than AUE

• On the one hand, when the dataset becomes higher-resolution and more category, the optimization difficulty in generating adversarial poisoning from scratch increases since AP finds non-robust features that rely on a good classifier.

• On the other hand, since supervised error minimization is easy to optimize, the high resolution provides more room for AUE to create effective poisoning patterns to fool the subsequent algorithms.

[W4] Contrastive learning and adversarial training If we understand correctly, the reviewer's question is why adversarial poisoning (AP) is effective for contrastive learning. We have discussed it in Appendix D.3 of our paper. On the one hand, [1] shows that some contrastive augmentation can squeeze low-frequency "shortcuts" in unlearnable examples. On the other hand, contrastive error minimization-based attacks such as CP and TP generate high-frequency perturbations (see Figure 8 in our paper). From the visualization, perturbations from AP contain more high-frequency patterns, which may deceive contrastive learning more easily.

[1] Liu Z, et al. Image shortcut squeezing: Countering perturbative availability poisons with compression. ICML 2023.

[Q1] Learning paradigms other than contrastive learning

• In Table 5 of our paper, we have checked the supervised contrastive learning (SupCL) and a semi-supervised method FixMatch.

• We evaluate our attacks against MAE by end-to-end fine-tuning on CIFAR-10. In the presence of AUE/AAP, the test accuracy of fine-tuning decreases by 57%/37%. In Figure 2 of the additional document, the training accuracy in the presence of attacks increases more quickly than that in the clean case, while the test accuracy is the opposite, meaning that the attacks create shortcuts for MAE. Although our methods were not specifically designed for MAE, they can transfer to MAE.

  Clean	AUE	AAP
  89.63	32.36	51.93

[Q2] Importance of data augmentation.

• Data augmentation is an essential component for CL since it creates different views of the same data point, which is fundamental for learning useful representations. Our work leverages stronger data augmentation to improve the effectiveness of SL-based attacks against CL.

• As other reviewers pointed out, there is a special case in CL, i.e., Text-Image contrastive learning algorithm CLIP. The image encoder of CLIP does not involve contrastive augmentation. We conduct experiments on linear probing upon the CLIP encoder and show that our attacks take effect on this algorithm. See our discussion with Reviewer 61jU for detailed results.

We would like to thank the reviewers for their valuable comments and suggestions! We hope our responses sufficiently address the concerns raised.

[Weakness1] Theoretical guarantees and why the enhancing data augmentation works.

• In fact, we conduct theoretical analysis for a toy model in the appendix to verify the intuition of our proposed method (Proposition E.1): if the supervised training process uses the same data augmentation as CL, then this process also optimizes an upper bound of the contrastive loss. As a result, supervised error minimization mimics contrastive error minimization. Such mimicry makes the modified SL-based generation process enhance the contrastive unlearnability of final perturbations.

• By the way, we agree with the reviewer that availability attacks currently lack theoretical guarantees similar to those provided by certified adversarial robustness or differential privacy. Although some works have clarified the optimization objectives of such attacks, it is still insufficient to guarantee the final achievement of unlearnability.

[W2] Technical contribution To obtain unlearnability for CL, we are the first to adopt contrastive augmentation in the SL-based training process. The proposed method is simple, effective, and efficient. Compared to other more complex algorithms, it is easier to scale to large datasets and has greater potential for real-world applications.

[Question] Biggest challenge In our opinion, the biggest challenge in this work is breaking free from the reliance on existing paths. Current methods on this topic, including CP, TUE, and TP, are all based on the contrastive training process which is time-consuming and meets challenges in scaling up to real-world datasets. We believe it’s possible to obtain effective availability attacks through the supervised training process which is more efficient and easier to optimize. The remaining task is to consider how to connect the characteristics of contrastive learning and supervised learning and propose an effective algorithm.

I appreciate the authors' response. My concerns regarding the theoretical analysis and technical contributions have been mostly addressed. I would like to raise my score accordingly.

We sincerely thank the reviewers for their thorough and constructive feedback! We aim to clarify and address your concerns through our detailed replies.

[Weakness1] Comparison with “One for All (14A)” We believe that "14A" and our work are more orthogonal in nature and will explain it in detail.

• The biggest difference between our method and “14A” is that we are studying different types of transferability. Specifically, “14A” studies cross-dataset transferability, meaning that the noise generator should produce UEs effective against supervised learning on different datasets. However, our paper investigates cross-algorithm transferability, meaning the generated UEs should be effective not only against SL but also against CL on the same dataset.

• “14A” does not claim to work for CL, nor does it present any experimental results related to CL. Meanwhile, we don't claim that our attacks transfer well across datasets since they are based on dataset-specific optimization instead of a noise generator and directly applying off-the-shell perturbations to a different dataset is much more difficult and less realistic.

[W2] Non-consistent advantage The reviewer mentioned that our proposed attacks do not consistently outperform baseline methods against different CL algorithms. We believe this phenomenon is not a weakness of our method; rather, it reflects the vulnerability of CL-based methods including CP, TUE, and TP, in terms of cross-algorithm transferability.

• CL-based attacks show varying performance against different CL algorithms (see Table 11 in our paper). For example, since TP is generated using SimCLR, it achieves rather low accuracy against SimCLR evaluation, i.e., 6.7% for CIFAR-100, while our AUE achieves 13.6%. However, when facing a different evaluation algorithm, saying BYOL, TP’s accuracy drastically increases to 27%, which is higher than AUE’s 19.2%.

• In contrast, our SL-based attacks demonstrate relatively stable performance against different CL algorithms and surpass CL-based baseline methods in worst-case unlearnability.

[W3] REM's augmentation The reviewer mentioned that our technique is an extension of the technique used by REM. We believe there are some misunderstandings here and we will elaborate below.

• “Expectation over transformation (EOT)” was proposed to make robust adversarial examples in the physical world[1], for example, a 3D-printed "adversarial" turtle that can be classified as a rifle from different views.

• Then REM uses a modified EOT in which the transformation distribution only contains Crop and Horizontal Flip. Note that these transformations are common in standard supervised learning and not exclusive to adversarial training. The ablation study empirically shows EOT can improve the performance of REM against adversarial training.

• We want to clarify that our method is not an EOT variant at all since it does not involve taking expectations. Moreover, our SL-based method contains contrastive augmentation that does not appear in SL. We leverage contrastive augmentation since it is a fundamental component of CL and our goal is to achieve unlearnability for CL.

• In summary, in terms of both motivation and technical details, our method is not an extension of EOT, but a technique specifically designed to address a particular problem.

[1] Athalye A, et al. Synthesizing robust adversarial examples. ICML 2018

[W5] Diffusion purification

• We acknowledge that at this stage of development, diffusion-based purification techniques have impacted all methods that aim to protect images through subtle perturbations, not only the method we studied, which uses availability attacks to prevent unauthorized data usage, but also methods that protect the copyrights of artists’ works [2,3,4].

• In Appendix D.9 of our paper, we discussed these techniques. We consider the defensive capability of diffusion purification as a limitation of this work, and exploring how to overcome this limitation is a promising direction for future research, such as incorporating the diffusion process into perturbation generation.

• Since the code in DiffAug's GitHub repository does not include implementations for commonly used datasets such as CIFAR and ImageNet, we needed to spend additional time modifying the code to adapt it to our problem. Once the results are available, we will update them in a subsequent version.

[2] Shan S, et al. Glaze: Protecting artists from style mimicry by {Text-to-Image} models. USENIX Security 2023

[3] Liang C, et al. Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples. ICML 2023

[4] Hönig R, et al. Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI.arXiv:2406.12027

[W6] Typo Thank you for pointing out it. We will correct it in the next version.

[Question] CLIP We conduct linear probing upon CLIP on CIFAR-10 following the official example in the GitHub repository. The CLIP encoder is pre-trained by OpenAI and fixed.

• The following results show that both AUE and AAP can make CLIP-extracted representations of training data deviate from the true representation distribution. Compared to AUE, AAP achieves more unlearnability in such scenarios.

  	Clean	AUE	AAP
  CIFAR-10	94.99	90.63	51.22
  CIFAR-100	80.00	72.56	66.82

• The reason for this could be that, CLIP exhibits adversarial vulnerability, making it easy to find pixel-level perturbations that alter feature semantics [5]. Although our attacks were not designed specifically for CLIP, the perturbations they generate might share some commonalities with CLIP’s adversarial examples. Compared to AUE, AAP, as an adversarial example for SL, possibly plays a greater role in confusing the feature semantics extracted by CLIP.

[5] Fort S. Adversarial examples for the OpenAI CLIP in its zero-shot classification regime and their semantic generalization

We thank the reviewers for their constructive feedback and valuable suggestions! We hope that our clarifications address your queries and concerns effectively.

[Weakness1] Topic novelty and previous works

• As the reviewer mentioned, the CP and TUE papers were the first to point out the vulnerability of unlearnable examples to contrastive learning.

• However, we notice that CL-based attacks are time-consuming and suffer the vulnerability of algorithm transferability, for example, CP is not consistently effective for SL and TP's performance decreases against different evaluation CL algorithms (see Section 5.2 in our paper).

• Regarding the challenges in efficiency and effectiveness, our work provides two SL-based attacks that achieve both SL and CL unlearnability.

• In summary, we sincerely accept your suggestions and will adopt a more modest tone in the abstract and introduction versions and will clarify our contributions in the subsequent.

[W2] Augmentation strategies We indeed conducted more ablation studies on augmentation strategies. Here are our results.

• (1)Dynamic augmentation scheme. In the paper, we implement the proposed attacks with a constant augmentation strength. We also attempt dynamic augmentation schemes including annealing and tempering which linearly change augmentation strength during the perturbation generation process. In Figure 1 of the additional document, the annealing scheme achieves better supervised unlearnability than the tempering scheme; regarding contrastive unlearnability, annealing outperforms tempering for small final strengths and just the opposite for large final strengths. Compared with the constant scheme in the paper, the tempering scheme with a particular final strength is marginally better, while other options are worse. In summary, these two dynamic schemes do not outperform the default constant scheme used in the paper.

• (2) Applying perturbation before/after augmentation Another key factor in the effectiveness of our method is that perturbation should be added before augmentation. In the perturbation generation process, if the perturbation is added before augmentation $\mu$, then its gradient propagates through $\mu$, i.e., $\nabla_\delta f(\mu(x+\delta))$; otherwise, its gradient does not go through $\mu$, i.e., $\nabla_\delta f(\mu(x)+\delta)$. We conduct an ablation study of applying perturbation after augmentation for AUE in Figure 3 of the additional document. In that case, enhancing augmentation strength does not improve the effectiveness against contrastive learning.

[Question1] Scenario Let’s continue the scenario described in [1]: Given that facial recognition tools can be trained by scraping people’s photos from the public web, can people still feel secure uploading their selfies to social media? Availability attacks (unlearnable examples) provide a tool that allows AI systems to ignore a person’s selfies; however, this is limited to AI systems based on supervised learning algorithms. Considering contrastive learning algorithms can currently achieve performance comparable to that of supervised learning, once AI systems begin to use contrastive learning, the protection of selfies collapses. This is why the effectiveness of attacks against contrastive learning in real-world scenarios is important. Our work provides an efficient method that is effective for both supervised learning and contrastive learning.

[1] Will Douglas. How to stop AI from recognizing your face in selfies. MIT Technology Review. https://www.technologyreview.com/2021/05/05/1024613/stop-ai-recognizing-your-face-selfies-machine-learning-facial-recognition-clearview/

[Q2] Augmentation strategies See our reply to [W2]