Adversarial Attacks on Data Attribution

Results for Shadow Attack:

Results for Outlier Attack:

Weakness 1: Black-box assumption of the threat model

We would like to clarify that while we described in total three assumptions in Section 3.2, the two attacks we proposed rely on different sets of assumptions.

As explicitly stated in Line 188-189, the Outlier Attack does NOT assume Assumption 2 (Access to data distribution and training algorithm), and it indeed does NOT require any information about the model other than query access. So Outlier Attack is indeed a black-box attack method.

It is only the Shadow Attack that requires information related to the training algorithm, including the rough model architecture. However, we have NEVER claimed that Shadow Attack is a black-box attack method in our paper.

To further clarify this point, we have added one sentence at Line 189 to explicitly state that Shadow Attack is a gray-box attack method while Outlier Attack is a black-box attack method.

Weakness 2: Creating outliers via mislabeling

We kindly refer the reviewer to section 5.2 of our paper. In the “ Data Component To Be Perturbed” paragraph, we have discussed why directly flipping the label of the training data point is not a practical attack strategy. To recap, the main reasons are:

1. Such an outlier with a wrong label often degrades model performance and is likely to be identified as negatively influential;
2. It is easy for the AI developer to detect such wrongly-labeled data points.

Weakness 3: OOD and adversarial defense against proposed attacks.

We appreciate the reviewer’s mention of OOD detection and adversarial defense techniques.

However, we respectfully disagree with the characterization of assuming no adversarial sample detection in the AI developer pipeline as "simplistic." There is an inherent tradeoff between robustness and utility in nearly all adversarial defense techniques. Without a clear understanding of potential threats, it is unlikely that AI developers will preemptively adopt adversarial defense strategies in their development pipeline.

To the best of our knowledge, this is the first systematic study of adversarial attacks on data attribution. Our study reveals effective yet surprising (especially the Outlier Attack) attack strategies on data attribution. It is precisely the key contribution of this work to alert AI developers that adversarial defense techniques are necessary when integrating data attribution into their pipeline. We also provide insights about how adversaries might exploit their system. This enables developers to select appropriate defense techniques without excessively comprimising model utility.

Additionally, we would like to highlight that the “outliers” generated by our attack methods may not resemble the natural outliers typically addressed by conventional OOD detection techniques. Therefore, the effectiveness of these methods in countering our proposed adversarial attacks remains uncertain.

While the development of effective defense techniques is an important avenue for future work, we believe that our study makes a significant contribution as the first systematic investigation of adversarial vulnerabilities in data attribution. By revealing surprisingly simple yet effective attack methods, including one requiring only black-box access to the model, this work lays a critical foundation for further research, even without an extensive exploration of defense strategies.

Weakness 4: Data augmentation defense against proposed attacks

We appreciate the reviewer’s suggestion of data augmentation as a simple defense against the proposed attacks. We acknowledge that data augmentation is a common step in AI development pipeline and have conducted additional experiments in such setups.

We conducted experiments on the image classification task where we applied cropping and affine-transformation augmentations. We assume that the adversary knows nothing about the augmentation and perturbs their own data as if there is no augmentation when they conduct shadow training in Shadow Attack or generate adversarial examples in Outlier Attack. The results are shown below and both attack methods are still highly effective under data augmentation.

Question 3: Attack performance under different sizes of $Z_0$ and $Z_1$.

We have conducted experiments for both attack methods with increased and decreased sizes of $Z_0$ and $Z_1$. The results can be found in Appendix C.1, Table 5 and 6. For convenience, we summarize the results here as well. We see that the results demonstrate the effectiveness of our methods across different sizes of the dataset. Overall, with a few exceptions, the Ratio of the Compensation Share appears to become even higher when the dataset size is larger.

The results of Shadow Attacks:

The results of Outlier Attacks:

Minor Edit: the 11 row 3 column of the result of Outlier Attacks in the initial post has a typo: the number $0.0031$ should be $0.0013$ which has been updated correctly now. Note that the Ratio value (11 row 5 column) is originally calculated correctly based on the correct value and remains accurate.

We thank you for your detailed insights and questions.

Weakness 1: The paper lacks well-defined research questions in evaluation.

Our experiment results primarily and effectively support the main argument: the data attribution score can be artificially increased by careful design of adversarial attack methods which we show in the paper. We have added a description at the beginning of Section 6 to clarify it.

Weakness 2: White-box attack can be added as another baseline.

We implemented the white-box attack where the adversary has full knowledge of the model on the image classification task. The results show that the effectiveness of the black-box attack is slightly worse but comparable to the white-box attack, further highlighting the effectiveness of the proposed methods.

Question 1: The distortion/perturbation budget of our attack methods.

We have closely followed the existing literature when setting the perturbation budget. Please check the relevant literature: ZOO [1], Simba [2], TextFooler [3]. We also had a detailed description in Appendix B.3 in our original submission.

[1] ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models. https://dl.acm.org/doi/abs/10.1145/3128572.3140448

[2] Simple Black-box Adversarial Attacks. https://arxiv.org/pdf/1905.07121

[3] Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. https://arxiv.org/pdf/1907.11932

Question 2: Experiment with different data attribution methods on the same setup.

We thank the reviewer for the suggestion. We have extended experiments with Influence Function and TRAK, in addition to the original Data Shapley, on experimental setup (b). Note that Data Shapley is much more computationally intensive than Influence Function and TRAK, which is not feasible on large datasets, especially under constrained computational resources. The results below show that the proposed attack methods are highly effective for all three data attribution methods on the same setup. The difference brought by different data attribution methods is relatively small in comparison to the absolute increase of compensation share.

The Results for Shadow Attack:

The Results for Outlier Attack:

We thank the reviewer for the acknowledgment of our work and the insightful comments.

Weakness 1: Justifying the surrogate objective.

The original compensation share objective measures the ratio that a specific subset of training data (contributed by the adversary) present in the top-k most influential set, while the surrogate objective measures the sum of the (proxy) influence scores of this training subset. Intuitively, these two objectives are aligned, while the surrogate objective is easier to optimize.

Furthermore, we highlight that in all of our experiments, the evaluation metric is the original compensation share, the fact that optimizing our surrogate objective leads to a good performance in the original compensation share metric suggests that our surrogate objective approximates the original compensation share reasonably well.

Weakness 2: Larger-scale experiments needed due to emphasis on LLM setup in Assumption 1

First, we would like to clarify that Assumption 1 only assumes that the training data collected from a previous time step will be largely reused in the next time step. While we mentioned LLMs and recommender systems as examples, this assumption is not specific to LLMs. This assumption may even better hold for smaller-scale applications where the training data are expensive to get, as in such cases, it becomes more likely for the data to be reused in the next time step.

Nevertheless, we agree that our paper could be further strengthened by larger-scale experiments. We have conducted a new experiment on the Tiny Imagenet dataset (see the table below), and we observe that both attack methods are still very effective.

We thank the reviewer for the insightful comments and questions.

We would like to first clarify a potential misinterpretation of our compensation mechanism. While the compensation mechanism indeed involved the data points with incorrect/undesired predictions, we highlight that the attribution score is calculated with respect to the loss on these data points, i.e., the top influential data points are always the ones contributing to the loss reduction (regardless the correctness of the predictions). Even when the model makes incorrect predictions on certain validation data points, the training data points that contribute to dragging the model towards the correct prediction direction could still be considered valuable.

To fully address the reviewer’s concern, we also conducted an experiment in which the AI developer would only reward the correct predictions on a private validation set on the image classification task. The result is presented as follows, and the increase in compensation share exceeds 200%, which shows the effectiveness of our method in a compensation mechanism where only correct predictions are attributed.

Results for Both Attacks:

The consistency between the two compensation mechanisms is not surprising, as both mechanisms aim to reward training data points that contribute to dragging the model towards the correct prediction direction.

Finally, we would like to highlight that in the Outlier Attack, the outliers are correctly labeled samples that do not look similar to the rest of the training dataset. Intuitively, this is why these outliers are more likely to be attributed as influential data points helping correct predictions.

Follow-Up Question: Does the attack not increase the target model performance/decrease loss (on the corresponding distribution or validation) when it is getting a bigger share of compensation?

We thank the reviewer for this insightful question. The attack does NOT increase the target model performance. Please see below a summary of the image classification accuracy across different settings.

In comparison to the model trained on the original clean data without manipulation, the model trained on the data manipulated by the attacks has similar or slightly worse performance in all settings.

Intuitively, the Compensation Share is a “zero-sum” metric among all the training data points. The attacks increase the Compensation Share of the training data points contributed by the adversary, which, by definition, means the Compensation Share of the rest of the training data points is decreased. The increase in Compensation Share of a subset of the training data does not necessarily lead to the overall performance gain.

We hope the additional results and the clarification on the intuition can successfully address your concern. We will also update the draft later to include this information.

Follow-Up Question on Model Performance

Thank you for raising this concern. Our experiments did not apply regularization techniques such as data augmentation, which led to a performance gap compared to the optimal one reported in the literature. The impact of such regularization techniques may be particularly important given the smaller training data size we used. To address your concern, we conducted an additional experiment with ResNet-18 on CIFAR-10 with $Z_0 = 147000$, $Z_1 = 150000$ (the full training data), with data augmentation. We only have results for Outlier Attack at this point, as Shadow Attack experiments take much more time. The results are summarized below.

We first observed that the accuracy of the model trained on the original data without manipulation is 93.8%, which is consistent with the literature. The model trained on the data after manipulation by the attack has slightly lower accuracy, which is consistent with our previous observations on smaller data settings. Meanwhile, the ratio of compensation share change shows that our method is even more effective in this standard ResNet training setup on CIFAR-10. We hope this can address your concern, and we will also update the draft to include this result.

We apologize for the confusion. We should have written $Z_0 = 49000$ and $Z_1=50000$. The 147k and 150k are the sizes after augmentation. We applied two data augmentations per sample (flipping + rotation and flipping + cropping), thus the total size was tripled. Below is the exact PyTorch function for our data augmentation:

The first augmentation: transforms.RandomVerticalFlip(), transforms.RandomRotation(15)

The second augmentation: transforms.RandomHorizontalFlip(), transforms.RandomCrop(32, padding=4)

Here is my confusion: I believe CIFAR-10 has only 60k samples in total (50k of which are training samples). Could the authors elaborate how you define a setting with 150k samples (i.e. z_0=147k & z_1=150k)?

Also providing details such as what augmentations are used would help.