Thank you for the valuable suggestions! I have submitted a revised draft addressing the comments provided by the reviewers. Below, you'll find the responses to the questions:

Q1. Concern regarding pseudo-robustness. If we design an adaptive poison pattern, that uses a white-box attack (e.g. PGD) to end-to-end (treat VAE and classifier as a whole) generate perturbations, can this method also be used for effective purification?

A1. This paper primarily addresses the defense against perturbative availability poisoning attacks, a type of data poisoning attack. Typically, such attacks occur during the data collection process or involve malicious modifications to the dataset before model training. Consequently, as the attacker has no access to model training, defense strategies against data poisoning attacks assume the trustworthiness of the defense process or model training process. As a result, it is generally not feasible for attackers to compromise the VAE during the purification process.

Q2. Concern regarding generalization capacity. If we use a new type of poisoning attack that has not been seen in class-wise embedding (out of the training poisoning attacks), can this method still effectively purify it?

A2. In research focusing on defenses against data poisoning attacks, the defense process is trustworthy, and attackers have no access after poisoning the training dataset. Once we obtains the poisoned dataset, we can proceed to train a D-VAE. Then, at inference time, the dataset to be purified has already been seen by D-VAE during training, eliminating the possibility of encountering a new type of poisoning attack that D-VAE has not been exposed to. The reviewer's suggested settings are indeed intriguing. However, given the constraints of the rebuttal timeline, we will consider and discuss these aspects in our future work.

Q3. Concern regarding training and inference costs. Does the training set of this method include all poisoning attacks? What is the cost of training? Three additional forward propagations are required in the algorithm, what is the inference time?

A3. The training dataset for our methods is the poisoned dataset, constructed using a single poisoning attack method. Consequently, for each poisoning attack, we train a distinct D-VAE for purification.

The table below presents the training time for D-VAE, the inference time for the poisoned dataset, and the time to train a classifier using the purified dataset. For comparison, we include the training-time defense Adversarial Training. It's important to note that the times are recorded using CIFAR-10 as the dataset, PyTorch as the platform, and a single Nvidia RTX 3090 as the GPU. As can see from the results, the total purification time is approximately one and a half times longer than training a classifier. Compared to adversarial training, our methods are about 5 times faster. Additionally, our method achieves an average performance around 90%, which is 15% higher than the performance achieved by adversarial training.

Dear Reviewer h4wy,

Thank you for taking the time to review our submission and providing us with constructive comments and a favorable recommendation. We would like to know if our responses adequately addressed your earlier concerns. Additionally, if you have any further concerns or suggestions, we would be more than happy to address and discuss them to enhance the quality of the paper. We eagerly await your response and look forward to hearing from you.

Best regards,

The authors

Q2. The role of the auxiliary decoder and the process of initializing and optimizing $\pmb{u_y}$, as well as optimizing $\pmb{u_y}$ to assist $D_{\theta_p}$ in generating the disentangled perturbation $\pmb{\hat{p}}$, are not explicitly explained. Additionally, the optimization of $\theta_p$ in the absence of groundtruth values for the added perturbations $\pmb{p}$ is unclear.

A2. We have revised and provided a detailed explanation of the components in Section 3.4, focusing on the learning process of disentangled perturbations.

• Initialization of $\pmb{u_y}$.

  • Given that the size of the input $\pmb{x}$ for each dataset is fixed, such as $3 \times 32 \times 32$ for the CIFAR-10 dataset, the size of the latents $\pmb{z}$ is fixed at $128 \times 16 \times 16$. Consequently, $\pmb{u_y}$ for each class is of the same size as $\pmb{z}$ and is defined and randomly initialized using torch.nn.Embedding when utilizing PyTorch as the platform.

• Optimize $\pmb{u_y}$ and $\theta_p$ in the absence of groundtruth values for the added perturbations $\pmb{p}$.

  • Given that the defender lacks groundtruth values for the perturbations $\pmb{p}$, it is not possible to optimize $\pmb{u_y}$ and $D_{\theta_p}$ to learn to predict $\pmb{\hat{p}}$ directly by minimizing ${\lVert \pmb{p} - \pmb{\hat{p}} \rVert}_2^2$ during model training.

  • Expanding on the insights from Section 3.2 and Remark 1 in Section 3.3, when imposing a low upper bound on the KLD loss, creating an information bottleneck on the latents $\pmb{z}$, the reconstructed $\pmb{\hat{x}}$ cannot achieve perfect reconstruction, making poison patterns more challenging to be recovered in $\pmb{\hat{x}}$. As a result, a significant portion of poison patterns $\pmb{p}$ persists in the residuals $\pmb{x} - \pmb{\hat{x}}$.

  • Following Proposition 2 in Section 3.3, the majority of poison patterns associated with each class data exhibit relatively low entropy, suggesting that they can be predominantly reconstructed using representations with limited capacity. Considering that most poison patterns are crafted to be sample-wise, we propose a learning approach that maps the summation of a trainable class-wise embedding $\pmb{u_y}$ and the latents $\pmb{z}$ to $\pmb{\hat{p}}$ through an auxiliary decoder $D_{\theta_p}$.

  • To learn $\pmb{u_y}$ and train $D_{\theta_p}$, we propose minimizing ${\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2$, as the residuals $\pmb{x} - \pmb{\hat{x}}$ contain the majority of the groundtruth $\pmb{p}$ when imposing a low upper bound on the KLD loss.

• Next, let's analyze why ${\lVert \pmb{x} - \pmb{\hat{x}} \rVert}_2^2$ and ${\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2$ are not contradictory.

  • As there is a information bottleneck in the latents $\pmb{z}$ and $\pmb{u_y}$, we assume that the best-reconstructed images are represented by $\pmb{\tilde{x}}$, and the best-reconstructed perturbations are represented by $\pmb{\tilde{p}}$.

  • If some parts of $\pmb{\tilde{x}}$ are not recovered in $\pmb{\hat{x}}$, we denote $\pmb{\hat{x}} = \pmb{\tilde{x}} - \pmb{\Delta{x}}$.

  • As $\pmb{\hat{p}}$ is generated by $\pmb{u_y}+\pmb{z}$, the $\pmb{\Delta x}$ could be recovered in $\pmb{\hat{p}}$, leading to $\pmb{\hat{p}} = \pmb{\tilde{p}} + \pmb{\Delta{x}}$.

  • We can observe that ${\lVert \pmb{x} - \pmb{\hat{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2$ is minimized when $\pmb{{\Delta x}}= \pmb{0}$, as ${\lVert \pmb{x} - \pmb{\hat{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2 = {\lVert \pmb{x} - \pmb{\tilde{x}} + \pmb{\Delta{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\tilde{x}}) - \pmb{\tilde{p}} \rVert}_2^2$ .

  • When $\pmb{{\Delta x}}= \pmb{0}$, ${\lVert \pmb{x} - \pmb{\hat{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2= {\lVert \pmb{x} - \pmb{\tilde{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\tilde{x}}) - \pmb{\tilde{p}} \rVert}_2^2$ .

  • In the table below, we present quantitative results indicating that $\pmb{\hat{x}}$ reconstructed by VAE or D-VAE exhibits similar PSNR when the same upper bound for KLD loss is employed. From the PSNR between $\pmb{x}$ and $\pmb{\hat{x}}$+$\pmb{\hat{p}}$, we observe that $\pmb{\tilde{p}}$ can recover some parts of $\pmb{p}$, making $\pmb{\hat{x}} + \pmb{\hat{p}}$ closer to $\pmb{x}$.

    Model/KLD loss	2.0	2.5	3.0	3.5	4.0
    VAE: PSNR($\pmb{x},\pmb{\hat{x}}$)	25.19	26.14	27.02	27.67	28.27
    D-VAE: PSNR($\pmb{x},\pmb{\hat{x}}$)	25.07	26.12	26.97	27.54	28.19
    D-VAE: PSNR($\pmb{x},\pmb{\hat{x}}$+$\pmb{\hat{p}}$)	25.64	26.71	27.70	28.42	29.07

Thank you for the valuable suggestions! I have submitted a revised draft addressing the comments provided by the reviewers. Below, you'll find the responses to the questions:

Q1. Offer a more precise definition of the attacks and a more comprehensive discussion, including detailed information on the availability attacks in Section 2.

A1. We have extensively revised the Data Poisoning in Section 2, presenting a thorough discussion of other types of attacks related to perturbative availability poisoning attacks. Perturbative availability poisoning attacks aim to degrade the overall performance on validation and test dataset. The attacker has the capability to poison the entire training data using bounded perturbations, e.g., $\Vert \pmb{p} \Vert_{\infty} \le \frac{8}{255}$, while keeping the labels unchanged.

Q11. For the Gray and JPEG baselines, are similar transformations applied to the test images during evaluation? Otherwise, these approaches are likely at a disadvantage due to a domain gap.

A11. For the Grayscale, JPEG, and BDR baselines, we adhere to the original settings in ISS [1], applying these transformations only to the training images and not to the test images during evaluation. Here, we report the performance when the same transformations are applied to the test images in the table below. Note that we select the CIFAR-10 as the dataset. From the results in the table, we observe that adopting the same transformations to the test images does not significantly affect the performance, and our method consistently outperforms the baselines.

Q12. Ethics concerns.

A12. Thank you for the guidance. We have incorporated an ETHICS STATEMENT section following the main content of the paper. We also list the content here.

In summary, our paper presents a effective defense strategy against perturbative availability poisoning attacks, which aim to undermine the overall performance on validation and test datasets by introducing imperceptible perturbations to training examples with accurate labels. Perturbative availability poisoning attacks are viewed as a promising avenue for data protection, particularly to thwart unauthorized use of data that may contain proprietary or sensitive information. However, these protective methods pose challenges to data exploiters who may interpret them as potential threats to a company's commercial interests. Consequently, our method can be employed for both positive usage, such as neutralizing malicious data within a training set, and negative purpose, including thwarting attempts at preserving data privacy. Our proposed method not only serves as a powerful defense mechanism but also holds the potential to be a benchmark for evaluating existing attack methods. We believe that our paper contributes to raising awareness about the vulnerability of current data protection techniques employing perturbative availability poisoning attacks. This, in turn, should stimulate further research towards developing more reliable and trustworthy data protection techniques.

Q7. Concerns regarding writings.

A7. Thank you once again for your thorough review of our paper. I have revisited the content and addressed all the mentioned concerns. The revised sections are highlighted in blue in the resubmitted draft.

Q8. Instead of a VAE, how does a non-variational auto-encoder with an information bottleneck work?

A8. In the newly added Appendix E, we conduct experiments on purification using non-variational auto-encoders (AEs) with an information bottleneck. To achieve non-variational auto-encoders with different bottleneck levels, we modify the width of the features within the auto-encoder architecture. This results in models with varying parameter numbers. Then, we proceed to train the AE on the poisoned CIFAR-10 dataset, and test on the clean test dataset with classifiers trained on the purified dataset. As illustrated in Figure 5 in the Appendix, a non-variational auto-encoder with an information bottleneck can also eliminate poison patterns. However, when comparing at a similar level of reconstruction quality measured by PSNR, VAEs demonstrate a stronger inclination to remove more poison patterns in both the REM and LSP poisoning attacks. For EM poisoning attacks, the outcomes are comparable. These observations align with the theoretical analysis presented in Section 3.3.

Q9. Why separately decode the perturbation and the reconstructed image? If the input $\pmb{x}$ is $\pmb{x_c}+\pmb{p}$, then knowing either $\pmb{x_c}$ or $\pmb{p}$ should yield the other, since $\pmb{x}$ is given.

A9. As observed in Section 3.2, setting a low upper bound on the KLD loss results in the reconstructed images $\pmb{\hat{x}}$ containing very few poison patterns but being heavily corrupted in terms of PSNR (e.g., PSNR is around 22 when the upper bound on the KLD loss is set to 1.0). Consequently, the residuals $\pmb{x} - \pmb{\hat{x}}$ contains both the majority of poison patterns and some parts of $\pmb{x_c}$ that cannot be reconstructed due to the information bottleneck in the latents $\pmb{z}$. Therefore, with an amplitude around 18 in terms of $\ell_2$-norm for $\pmb{x} - \pmb{\hat{x}}$ and 1.5 for $\pmb{p}$, attempting to estimate $\pmb{p}$ based on $\pmb{x} - \pmb{\hat{x}}$ will result in a significant error.

At the same time, since the decoder for generating the disentangled perturbations $\pmb{\hat{p}}$ is trained in an unsupervised manner by minimizing ${\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2$. Considering the bottleneck in $\pmb{z} + \pmb{u_y}$, $\pmb{\hat{p}}$ cannot be fully reconstructed and typically represents only parts of the groundtruth perturbations $\pmb{p}$. Consequently, although $\pmb{x}-\pmb{\hat{p}}=\pmb{x_c}+(\pmb{p}-\pmb{\hat{p}})$ contains fewer poison patterns than $\pmb{x}$, the remaining patterns can still be effective for poisoning.

Based on the aforementioned concept, we employ a two-stage purification framework in Section 3.5. In the first stage, we utilize a small KLD limit and retain $\pmb{x}-\pmb{\hat{p}}$, containing fewer poison patterns. In the second stage, a large KLD limit is applied, preserving $\pmb{\hat{x}}$, which exhibits good reconstruction quality and retains most useful features for classification.

Q10. What percent of the data is poisoned in the main experiments in Tables 2-5?

A10. Our paper mainly addresses defenses against perturbative availability poisoning attacks, where the entire training data is poisoned with bounded perturbations. Consequently, in the main experiments presented in Tables 2-5, the poisoning rate is consistently set to 100%.

Q3. Model Trainer Inspection of Training Data: the model trainer can inspect training data as the model performs worse on the test data.

A3. In certain vertical domain, the attacker is capable of poisoning the entire training dataset. The model trainer will lack clean data for validation and might remain unaware of the poisoning, rendering any inspection of the data futile.

Q4. Detection of Poisoned Data: Can the poisoned data be detected more simply?

A4. In the case of perturbative availability poisoning attacks, where the entire training dataset is poisoned, there is limited research on detecting poisoned data compared to backdoor attacks. Perturbative availability poisoning attacks also exists on attacking large foundation models, such as Anti-Dreambooth [5] that also poisons all the training samples used for fine-tuning the diffusion models. In section 5, we also demonstrate that our proposed method can be utilized to detect poisoned data, a aspect not previously explored in related research that works on defenses [1].

Q5. Computation requirements for the purification process, and additional requirements for hyperparameters. Comparison with JPEG compression.

A5. The table below presents the training time for D-VAE, the inference time for the poisoned dataset, and the time to train a classifier using the purified dataset. For comparison, we include the training-time defense Adversarial Training. It's important to note that the times are recorded using CIFAR-10 as the dataset, PyTorch as the platform, and a single Nvidia RTX 3090 as the GPU. As can see from the results, the total purification time is approximately one and a half times longer than training a classifier. Compared to adversarial training, our methods are about 5 times faster. Additionally, our method achieves an average performance around 90%, which is 15% higher than the performance achieved by adversarial training.

For the second stage, which involves training D-VAE, the hyperparameters, including $kld_{limit}$ and downsample size, are consistently set to the same values for all datasets (CIFAR-10, CIFAR-100, and ImageNet-subset) across various attack methods and perturbation levels. While in the first stage which aims to recover most parts of poison patterns, the hyperparameters differs mostly due to different input size. Since ImageNet-subset comprises larger images (size $224 \times 224$) compared to CIFAR-10, to enhance the disentanglement of poison patterns, we employ a deeper downsampling in the experiments on ImageNet-subset.

Despite JPEG compression being a more efficient defense method, its performance is lower than our methods, with an 18% lower on CIFAR-10 and a 9% lower on CIFAR-100 and ImageNet-subset. Considering the time taken for the standard training of the classifier, the total time required by our methods is about two and a half times that of JPEG, which is still within an acceptable range.

Q6. Technical notations.

A6. We have now fixed all the notations and standardized all the symbols in the updated manuscript.

• $kld_{limit}$ is the upper bound on the KLD loss.

• $\pmb{z}$ is the latent features from the encoder in VAE/D-VAE, which is resampled from $\mathcal{N}(\mu,\sigma)$, and $\mu,\sigma$ are the direct outputs from the encoder.

• $\pmb{x_c}$ is the clean data, $\pmb{p}$ is the added perturbations, $\pmb{x}$ is the poisoned data, $y$ is the label.

• $\mathcal{D}$ is the clean test dataset, $\mathcal{T}$ is the clean training dataset, $\mathcal{P}$ is the poisoned training dataset.

• $\pmb{\hat{x}}$ is the reconstructed data by VAEs/D-VAEs, $\pmb{\hat{p}}$ is the disentangled perturbations by D-VAEs, $\pmb{z}$ is the latents encoded by the VAEs/D-VAEs.

• D-VAE consists of encoder $E_{\phi}$, decoder $D_{\theta_c}$, auxiliary decoder $D_{\theta_p}$, and class-wise embeddings ${\pmb{u_y}}$, which are all trainable.

• For the theoretical analysis in Section 3.3, we formulate expressions in the feature space, and thus we now change $\pmb{x}$ to $\pmb{v}$ to denote the features. And $\pmb{v}=(\pmb{v_c},\pmb{v_s^t})$ denote the paired features extracted from the data. $t$ is used to denote that the feature is non-predictive.

[5] Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc Tran and Anh Tran. Anti-DreamBooth: Protecting Users from Personalized Text-to-Image Synthesis. In ICCV, 2023.

Thank you for the valuable suggestions! I have submitted a revised draft addressing the comments provided by the reviewers. Below, you'll find the responses to the questions:

Q1. Questions regarding the disentangled poison patterns in the Strengths part.

A1. Given that the defender lacks groundtruth values for the perturbations $\pmb{p}$, it is not possible to optimize $\pmb{u_y}$ and $D_{\theta_p}$ to learn to predict $\pmb{\hat{p}}$ directly by minimizing ${\lVert \pmb{p} - \pmb{\hat{p}} \rVert}_2^2$ during model training. Instead, as the residuals $\pmb{x} - \pmb{\hat{x}}$ contain the majority of the groundtruth $\pmb{p}$ when imposing a low upper bound on the KLD loss, we propose minimizing ${\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2$. As $\pmb{\hat{p}}$ is generated by $\pmb{u_y}+\pmb{z}$, which has an information bottleneck, it is hard to achieve a perfect reconstruction of $\pmb{p}$, and $\pmb{\hat{p}}$ is most likely to be a part of $\pmb{p}$. In the table below, we offer the $\ell_2$-norm of both $\pmb{p}$ and $\pmb{\hat{p}}$ , and we can see that the $\pmb{\hat{p}}$ has a smaller amplitude. In Section 4.2, the experiments show that the $\pmb{\hat{p}}$ remains effective as poison patterns. Notably, the amplitude of $\pmb{\hat{p}}$ is comparable to that of $\pmb{{p}}$, with $\pmb{\hat{p}}$ being slightly smaller than $\pmb{{p}}$ except for OPS.

Visual results of the normalized perturbations are in Figure 1, and we observe a striking similarity between $\pmb{\hat{p}}$ and $\pmb{{p}}$, especially for LSP and OPS. Since LSP and OPS use class-wise perturbations (i.e., perturbations are identical for each class of images), they exhibit lower class-conditioned entropy compared to other attack methods that employ sample-wise perturbations. This makes the reconstruction of LSP and OPS perturbations much easier.

Q2. Ratio of Poisoned Data is Unpractical: it is not realistic to poison the whole training data.

A2. Our paper mainly addresses defenses against perturbative availability poisoning attacks. We have revised the "Data Poisoning" section in the related works to explicitly outline the attack setting and distinguish it from other related attacks, e.g., backdoor attacks, and some types of poisoning attacks.

Availability attacks aim to degrade the overall performance on validation and test datasets by solely poisoning the training dataset. Typically, such attacks inject poisoned data into the clean training set. Poisoned samples are usually generated by adding unbounded perturbations, and take only a fraction of the entire dataset [2]. The performance degradation induced by these attacks is relatively not high at a small poison ratio (e.g., a 16% drop on CIFAR-10 [3]), and the poisoned samples are relatively distinguishable [3].

Another recent emerging type is perturbative availability poisoning attacks [1], where samples from the entire training dataset undergo subtle modifications (e.g., bounded perturbations $\Vert \pmb{p} \Vert_{\infty} \le \frac{8}{255}$), and are correctly labeled. This type of attack, also known as unlearnable samples [4], can be viewed as a promising approach for data protection. Models trained on such datasets often approach random guessing performance on clean test data.

Our paper mainly focuses on defenses against perturbative availability poisoning attacks, since defense against these attacks remains a challenging problem, e.g., existing data sanitization techniques are not applicable as the entire dataset is perturbed. Therefore, unless explicitly stated otherwise, the poisoning rate in our paper is set to 100%. This choice allows us to showcase the effectiveness of the proposed defense method. In line with previous researches on perturbative availability poisoning attacks [3], we also showcase the effectiveness of our proposed methods in partial poisoning settings, i.e., taking a lower poisoning rate.

[1] Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Image shortcut squeezing: Countering perturbative availability poisons with compression. In ICML, 2023

[2] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017

[3] Yiwei Lu, Gautam Kamath, and Yaoliang Yu. Exploring the limits of model-targeted indiscriminate data poisoning attacks. In ICML, 2023.

[4] Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. In ICLR, 2021.

Q3. Computation and comparison with JPEG.

A3. Thank you for the suggestions. We add a discussion on computation and a comparison with JPEG in the Appendix F. We also note a limitation in the JPEG compression approach used in ISS [1]—specifically, they set the JPEG quality to 10 to purify poisoned samples, resulting in significant image degradation. In the table below, we present results using JPEG with various quality settings. We also include the PSNR between purified images and input images in the table. Notably, our proposed methods consistently outperform JPEG compression when applied at a similar level of image corruption.

As evident from the observations in Section 3.2 and the theoretical analysis in Section 3.3 of the main paper, the effectiveness of JPEG compression may be largely due to its severe degradation on the poisoned images, while VAEs demonstrate a tendency to remove poison patterns. Therefore, JPEG may exhibit suboptimal performance in the presence of larger perturbation bounds. In contrast, our method excels in eliminating the majority of poison patterns in the first stage, rendering it more robust to larger perturbation bounds. Table 5 in Section 4.3 of the main paper illustrates that when confronted with LSP attacks with larger bounds, our method demonstrates significantly smaller performance degradation compared to JPEG (with quality 10), e.g., 86.13% Vs. 41.41% in terms of test accuracy.

Thanks authors, for the response. I’ve read them, along with the other reviews.

Disentangled poison patterns.

I’m still a little confused why we want to minimize $||(x-\hat{x}) - \hat{p}||_2^2$. If we learn this perfectly, then we predict a $\hat{p}$ that we could have gotten simply by subtracting our cleaned prediction $\hat{x}$ from the input $x$. I also stand by my original comments that the reconstructed poisons are not actually that close, and not what I would describe as “striking”. Amplitude is not enough for comparing the noise patterns. What is the reconstruction error, relative to the noise amplitude? Or PSNR?

Ratio of Poisoned Data is Unpractical

I’m still not convinced on this point. In the scenario that is described in the second paragraph of the Introduction (scraping data from the Internet), it seems incredibly unlikely that all the data that one would scrape would be poisoned. The only scenario that I can currently envision then is if someone or some entity has specifically poisoned a dataset (almost as a form of encryption), and we then try to train a model on it. In that case, a) why would the poisoned dataset be available to us in the first place, and b) why would want to train on it? Regardless, this seems like a situation where we shouldn’t be trying to train on the poisoned dataset in the first place, as we’re clearly going against the dataset owner’s wishes, or training on something that should be recognizable as something designed to produce garbage results--in which case we should just go find other data.

Computation

Thank you for providing this. I encourage the authors to include this discussion in their paper. I somewhat agree with the authors that data purification taking 1.5x longer than training may be acceptable, especially since this should hopefully only have to be done once, but this is still a disadvantage worth acknowledging, especially since the JPEG approach is already “baked in” to virtually every training pipeline and does provide competitive protection.

Ethics Statement

Thanks for adding this to the paper.

Thank you for the additional responses and updates to the paper.

I understood that $\hat{p}$ is not available as an optimization target, but the proposed alternative is not completely satisfactory either. $\hat{p}$ should be directly producible from a correct cleaned image prediction $\hat{x}$, and because both $\hat{x}$ and $\hat{p}$ are model outputs, it's hard to say how optimization will choose to minimize this objective: a sizable component of the perturbation may end up being learned into $\hat{x}$.

Regardless, due to the other clarifications and improvements, I may be willing to upgrade my score to 6, but I still feel this paper is somewhat borderline due to the overall premise and methodology.

Q3. Are the first two items in Eq. 14 to train the D-VAE contradictory?

A3. We have revised and provided a detailed explanation of the components in Section 3.4, focusing on the learning process of disentangled perturbations. In conclusion, the two terms are not contradictory as both the reconstruction of $\pmb{\hat{x}}$ and $\pmb{\hat{p}}$ have an information bottleneck.

• A comprehensive explanation of the loss design for training D-VAE.

  • Given that the defender lacks groundtruth values for the perturbations $\pmb{p}$, it is not possible to optimize $\pmb{u_y}$ and $D_{\theta_p}$ to learn to predict $\pmb{\hat{p}}$ directly by minimizing ${\lVert \pmb{p} - \pmb{\hat{p}} \rVert}_2^2$ during model training.

  • Expanding on the insights from Section 3.2 and Remark 1 in Section 3.3, when imposing a low upper bound on the KLD loss, creating an information bottleneck on the latents $\pmb{z}$, the reconstructed $\pmb{\hat{x}}$ cannot achieve perfect reconstruction, making poison patterns more challenging to be recovered in $\pmb{\hat{x}}$. As a result, a significant portion of poison patterns $\pmb{p}$ persists in the residuals $\pmb{x} - \pmb{\hat{x}}$.

  • Following Proposition 2 in Section 3.3, the majority of poison patterns associated with each class data exhibit relatively low entropy, suggesting that they can be predominantly reconstructed using representations with limited capacity. Considering that most poison patterns are crafted to be sample-wise, we propose a learning approach that maps the summation of a trainable class-wise embedding $\pmb{u_y}$ and the latents $\pmb{z}$ to $\pmb{\hat{p}}$ through an auxiliary decoder $D_{\theta_p}$.

  • To learn $\pmb{u_y}$ and train $D_{\theta_p}$, we propose minimizing ${\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2$, as the residuals $\pmb{x} - \pmb{\hat{x}}$ contain the majority of the groundtruth $\pmb{p}$ when imposing a low upper bound on the KLD loss.

• Next, let's analyze why ${\lVert \pmb{x} - \pmb{\hat{x}} \rVert}_2^2$ and ${\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2$ are not contradictory.

  • As there is a information bottleneck in the latents $\pmb{z}$ and $\pmb{u_y}$, we assume that the best-reconstructed images are represented by $\pmb{\tilde{x}}$, and the best-reconstructed perturbations are represented by $\pmb{\tilde{p}}$.

  • If some parts of $\pmb{\tilde{x}}$ are not recovered in $\pmb{\hat{x}}$, we denote $\pmb{\hat{x}} = \pmb{\tilde{x}} - \pmb{\Delta{x}}$.

  • As $\pmb{\hat{p}}$ is generated by $\pmb{u_y}+\pmb{z}$, the $\pmb{\Delta x}$ could be recovered in $\pmb{\hat{p}}$, leading to $\pmb{\hat{p}} = \pmb{\tilde{p}} + \pmb{\Delta{x}}$.

  • We can observe that ${\lVert \pmb{x} - \pmb{\hat{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2$ is minimized when $\pmb{{\Delta x}}= \pmb{0}$, as ${\lVert \pmb{x} - \pmb{\hat{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2 = {\lVert \pmb{x} - \pmb{\tilde{x}} + \pmb{\Delta{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\tilde{x}}) - \pmb{\tilde{p}} \rVert}_2^2$ .

  • When $\pmb{{\Delta x}}= \pmb{0}$, ${\lVert \pmb{x} - \pmb{\hat{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\hat{x}}) - \pmb{\hat{p}} \rVert}_2^2= {\lVert \pmb{x} - \pmb{\tilde{x}} \rVert}_2^2 + {\lVert (\pmb{x} - \pmb{\tilde{x}}) - \pmb{\tilde{p}} \rVert}_2^2$ .

  • In the table below, we present quantitative results indicating that $\pmb{\hat{x}}$ reconstructed by VAE or D-VAE exhibits similar PSNR when the same upper bound for KLD loss is employed. From the PSNR between $\pmb{x}$ and $\pmb{\hat{x}}$+$\pmb{\hat{p}}$, we observe that $\pmb{\tilde{p}}$ can recover some parts of $\pmb{p}$, making $\pmb{\hat{x}} + \pmb{\hat{p}}$ closer to $\pmb{x}$.

    Model/KLD loss	2.0	2.5	3.0	3.5	4.0
    VAE: PSNR($\pmb{x},\pmb{\hat{x}}$)	25.19	26.14	27.02	27.67	28.27
    D-VAE: PSNR($\pmb{x},\pmb{\hat{x}}$)	25.07	26.12	26.97	27.54	28.19
    D-VAE: PSNR($\pmb{x},\pmb{\hat{x}}$+$\pmb{\hat{p}}$)	25.64	26.71	27.70	28.42	29.07

[1] Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Image shortcut squeezing: Countering perturbative availability poisons with compression. Proc. Int’l Conf. Machine Learning, 2023

Thank you for the valuable suggestions! I have submitted a revised draft addressing the comments provided by the reviewers. Below, you'll find the responses to the questions:

Q1. Math symbols should be unified.

A1. We have now standardized all the symbols in the updated manuscript.

• $\pmb{x_c}$ is the clean data, $\pmb{p}$ is the added perturbations, $\pmb{x}$ is the poisoned data, $y$ is the label.
• $\mathcal{D}$ is the clean test dataset, $\mathcal{T}$ is the clean training dataset, $\mathcal{P}$ is the poisoned training dataset.
• $\pmb{\hat{x}}$ is the reconstructed data by VAEs/D-VAEs, $\pmb{\hat{p}}$ is the disentangled perturbations by D-VAEs, $\pmb{z}$ is the latents encoded by the VAEs/D-VAEs.
• D-VAE consists of encoder $E_{\phi}$, decoder $D_{\theta_c}$, auxiliary decoder $D_{\theta_p}$, and class-wise embeddings ${\pmb{u_y}}$, which are all trainable.

Q2. More information in Section 3.2 and a detailed description and analysis of Figure 2.

A2. We have revised the majority of Section 3.2, providing a comprehensive description and analysis.

• Among earlier works, ISS [1] suggests the use of JPEG compression as an effective method to eliminate poisoned patterns. The central idea discussed in Section 3.2 aims to illustrate that VAEs tend to remove more poison patterns compared to JPEG compression. To investigate this, we employ EM, REM, or LSP as the attack methods to construct the poisoned dataset.
• We proceed to train VAEs with varying upper bounds for the KLD loss on the poisoned dataset $\mathcal{P}$. Following this, we report the accuracy on the clean test set $\mathcal{D}$ achieved by a ResNet-18 trained on the reconstructed data $\pmb{\hat{x}}$. In the left image of Figure 2(a), It is observed that reducing the KLD loss leads to a decrease in reconstruction quality, measured by the PSNR between $\pmb{\hat{x}}$ and $\pmb{{x}}$. Additionally, this reduction eliminates added poison patterns and original valuable features. The right image of Figure 2(a) shows that the increased removal of poison patterns in $\pmb{\hat{x}}$ correlates with improved test accuracy. However, when $\pmb{\hat{x}}$ is heavily corrupted, further reducing the KLD loss removes more valuable features, leading to a drop in test accuracy.
• In Figure 2(b), we contrast the outcomes of VAEs with JPEG compression at different quality settings. Across all results, when processed through VAEs and JPEG to achieve similar PSNR, the test accuracy with VAEs surpasses that of JPEG. This suggests that VAEs are significantly more effective at eliminating poison patterns than JPEG compression, when achieving similar levels of reconstruction quality.

[1] Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Image shortcut squeezing: Countering perturbative availability poisons with compression. Proc. Int’l Conf. Machine Learning, 2023

Dear Reviewer kPQt,

Thank you for taking the time to review our submission and providing us with constructive comments and a favorable recommendation. We would like to know if our responses adequately addressed your earlier concerns. Additionally, if you have any further concerns or suggestions, we would be more than happy to address and discuss them to enhance the quality of the paper. We eagerly await your response and look forward to hearing from you.

Best regards,

The authors