Thank you for reviewing our paper and we would like to address your concerns below.

The authors should first describe the framework for systematizing data poisoning and defenses, then introduce the related work by matching the framework with the existing methods. This would increase the clarity of the paper, that otherwise is confusing and seems disconnected in its sections.

We appreciate the reviewer's suggestion. We have updated the paper with revisions highlighted in blue to improve the clarity of the paper. Regarding the organization of the paper, we incorporated the problem definition in Section 3, which formally defines the APA attackers and defenders, into Sections 2.1 and 2.2 respectively. We believe this improves the flow of the paper and aligns with the reviewer's suggestion. We also welcome any further suggestions to improve the clarity of the paper.

"This work proposes a benchmark of data poisoning attacks. The authors suggest an unified codebase where to test poisoning attacks and defenses." "Does not compare with existing benchmarks / limited contribution."

Thank you for referencing the related works, and we have expanded the related data poisoning benchmarks and frameworks in Section 2.3.

We believe there could be a misunderstanding regarding the purpose of availability poisoning attacks and defenses.

We kindly point out that while APAs can be categorized as data poisoning attacks, there is a notable distinction between APAs and traditional data poisoning attacks regarding the threat model and the objectives. The purpose of APAs is to protect the privacy of the attacker's public data against unauthorized training. The goal of APAs is thus to make it difficult for deep learning algorithms to learn effectively from the data protected by APAs. The intent of APAs is therefore benign rather than malicious as generally assumed of data poisoning attacks. For this reason, while traditional data poisoning aims to preserve clean accuracies and inject backdoors, models trained with availability poisons are expected to have low clean accuracies in order to achieve the goal of data privacy protection.

In Section 2, we have expanded the problem definition (previously in Section 3) to explain the problem more clearly, and further clarify the differences between the two.

As the referenced survey and benchmarks are for traditional data poisoning attacks, which are malicious in nature and aims to achieve backdoors while preserving clean accuracy, we believe they unfortunately cannot be directly implemented as APAs, and compared with existing APAs.

Additional weaknesses: limitations are not discussed, and the paper exceeds the page limit

We kindly point out that Appendix D discusses the limitations of APBench and also per ICLR 2024 Author Guide, "This optional ethic statement/reproducibility statement will not count toward the page limit, but should not be more than 1 page".

Thank you for reviewing our paper and we would like to address your concerns below.

Clarifications on the threat model.

Since APAs can be categorized as data poisoning attacks, we follow existing naming conventions which refer to the "attacker" as the one that adds the perturbations, and the "defender" as the one that trains the model. In that sense, "poisoning" actually refers to the attacker adding perturbations to the data to hinder the defender's ability to train models.

With the above definitions, we can summarize the threat model as follows: We assume that the attacker has a set of data which they wish to publish while protecting them against machine learning. By introducing small perturbations into the data, the attacker aims to hinder the trainer's (defender's) ability to train models that can generalize effectively to the original data distribution from the protected data.

While both traditional data poisoning attacks and APAs add small perturbations to the data, the intention of the APAs is not to introduce stealth backdoor behaviors in models while preserving clean accuracies, but rather to reduce the ability of the models to learn from the data with crafted perturbations, in order to protect the attacker's privacy.

We have also since expanded Section 3 to explain the threat model in greater detail, and further clarify the differences between the related benchmarks in Section 2.

I suggest highlighting useful text to help readers understand what in the final conclusion to draw from an eval.

At least some of the evaluations should highlight how APBench will help a practitioner that prior works alone cannot identify.

Thank you for your constructive suggestions! We have highlighted the valuable insights with italics in the updated version. We also kindly point out that the main key takeaway of APBench is it is the first benchmark to shed light on the inadequacy of existing APAs against stronger defenses.

"As can be seen in Figure 3 ... the whole dataset."

Thank you for noticing this! We meant to claim that "the test accuracy of the model in the case of partial poisoning is only slightly lower than that in the case of a completely clean dataset," instead of "poisoning the whole dataset." The paper has been updated to reflect this.

Mean losses of Figure 4: are these over training steps/ samples?

For the mean losses, we take the final model after training, and evaluate the mean loss values by averaging the loss values of the model for each image in the respective datasets.

Can you rephrase the question: "can the protective perturbation… model training?"

We agree with the reviewer the question can be rephrase be more readable. For this we have updated the question to be as follows in the paper: "Are APAs effective in protecting only a portion of the training data? To answer, we introduce poisoning perturbations with APAs to a varying portion of the training data, and investigate how well the models learn the origin features that exist in the poisoned images for different poisoning rates."

If the private data is not supposed to be learned, why is it in the training data in the first place?

Quoting from [1], "Personal data has been unconsciously collected from the Internet and used to train commercial models, raising public concerns about the 'free' exploration of personal data for unauthorized or even illegal purposes." As none of us are immune to having our private data collected, perturbations can be added with APAs in order to protect data against unauthorized training.

[1] Huang, Hanxun, et al. Unlearnable examples: Making personal data unexploitable, 2021

Future methods should enhance the resilience of perturbation: I am not sure if this is for good (improve privacy) or bad (improve attacks)? Clarify.

We hope it is clear by now that APAs are meant to protect data against unauthorized training. Defenses explored in this paper are used to evaluate the effectiveness of APAs.

Thank you for reviewing our paper and acknowledging its contribution. We would like to address your concerns below.

defenses that have been developed for resisting backdoor attacks might be capable of mitigating the existing availability attacks

How might defenses originally developed for backdoor attacks, e.g. ANP [3], be effectively repurposed to mitigate availability attacks?

Thanks for your suggestion! We agree with the reviewer that backdoor attacks may present a promising avenue to mitigate availability attacks. However, due to differences in the objectives, most of the current backdoor defense methods require a certain number of clean samples (including ANP as kindly suggested by the reviewer), which may subjectively violate the assumed capabilities of the defender.

The TAP attack should be partially attributed to Nakkiran.

Thanks for pointing this out, we have updated this description in the updated version.

How about their test robustness, e.g., PGD-20 accuracy?

Thank you for the suggestion, and we have added this evaluation in Table 14 of Appendix C.2 of the updated version.

Thank you for reviewing our paper and we would like to address your concerns below.

Inadequate explanation for attack selection.

We kindly point out that while APAs can be categorized as data poisoning attacks, there is a notable distinction between APAs and traditional data poisoning attacks ([1], [2], [4]) regarding the threat model and the objectives. The goal of APAs is to make it difficult for deep learning algorithms to learn effectively from the data protected by APAs. The intent of APAs is therefore benign rather than malicious as generally assumed of data poisoning attacks.

In contrast, traditional data poisoning injects triggers into a small amount of data in a dataset, causing any input with the injected trigger to be misclassified by a model trained on the poisoned dataset.

Perhaps the reviewer could explain why "the assumptions of availability poisoning attacks ... do not assume clean labels". In general, existing APAs assume the training party has the ability to curate the correct labels for all images.

In Section 2, we have expanded the problem definition (previously in Section 3) to explain the problem more clearly, and further clarify the differences between the two.

Setup lacks explanation.

Thank you for your raising this question. We consider attackers who can initially generate poisons using all clean data and then select a subset of these generated poisons. For APA attacks that require a surrogate model, we assume that the attackers have access to the full training data, to make the attacks stronger and to also compare them fairly. As for the surrogate-free APA attacks (AR and LSP), no training data is required as they generate strongly linear separable perturbations algorithmically.

Inconsistent results.

Thank you for noticing this! We found that Figure 3 mistakenly repeats the Grayscale results for JPEG defenses, and have fixed this mistake in the updated version.

Conclusions that appear to be at odds with the available evidence.

Again, thank you for noticing this! We fully agree with the reviewer, and we meant to claim that "the test accuracy of the model in the case of partial poisoning is only slightly lower than that in the case of a completely clean dataset," instead of "poisoning the whole dataset."

These defense mechanisms have been explored in previous work like [5] and [6] and should be given more attention.

Thank you for your kind suggestion. [5] explores a similar setting as the HYPO attack examined in this paper.

We have now included early stopping (ES) as considered in [6] in the updated paper as a training-phase defense, and we report the peak accuracies achieved in Tables 3, 4, 5 and 6. We found that as a defense it is not very effective against most APA attacks in comparison to other defenses.