Influence-based Attributions can be Manipulated

Dear Reviewer,

Thanks for taking the time to review our paper, we deeply appreciate it.

We are very glad that you find the idea of exposing vulnerabilities in influence functions interesting. Next we address your concerns.

a) “these variations should not be considered an attack on the influence function” : What is an attack? An attack is a tool for an adversary to achieve its desired goal without getting caught. Attacks are meant to exploit some weakness or vulnerability otherwise how can a perfectly secure system be attacked? One of the vulnerabilities exploited by our proposed attack is the sensitivity of influence functions to model parameters – but this is no grounds to not call the method an attack! Infact our paper serves as a cautionary tale to not use sensitive methods (such as influence functions) for consequential tasks such as data valuation!

b) Reasonable attack scenarios : Perhaps there are some misunderstandings, we clarify them one by one.

“why would an adversary be able to modify the model parameters?”: Please note that the adversary in our case, influence calculator, is also responsible for training the model. For the attack, the adversary makes a few updates starting with the original honest model, which will be possible even for large models. Think of it like a company training some model and outputting both the influence scores (for compensating data providers) and a final model (which can be used by other entities such as other teams or customers).

“why would the evaluator choose a malicious model over a set of trusted models”: In the absence of a clear idea of what is an ‘evaluator’ here is our best explanation : The setting where there exists a separate evaluator with access to multiple models is a totally different threat model and should be a different paper. While interesting, there could be multiple threat models and there is no one universal threat model which is applicable in all settings. Different circumstances call for different threat models. We work on one setting and propose one threat model. If you have any concerns on the threat model used in our paper, we would be happy to clarify them.

c) Contributions of our paper: Note that ours is the first paper which introduces the problem of attacking attributions and proposes attacks with in-depth analysis. We believe the most important contribution of the paper is introducing and motivating the problem of attribution attacks – it is not the methodology or the results or the techniques (though they are very interesting in their own right) – this has opened the doors to future works such as attacking other attribution methods and more complex models. It wasn’t clear how the problem should be formulated or if there is a computationally feasible way of attacking influence functions or if the attacks will succeed even in simpler settings.

Additionally, we have extra contributions such as an impossibility theorem (Thm. 1) which points to the non-triviality of the problem and an efficient way of backpropagating through influence functions or inverse-hessian-vector-products – which is of independent technical interest.

d) Example for a real world scenario where our threat model is feasible : Note that the influence calculator is also compensating data providers. There could be two very natural scenarios for why it is the attacker 1) The influence calculator/model trainer is incentivized financially and wants to save money. Therefore it wants to artificially reduce the compensation it pays to the data providers, which can be done by reducing influence scores. Note that our proposed attacks are symmetric and can be used to both increase or decrease the data value. 2) Some data providers have colluded with the influence calculator to increase their own compensation (while reducing that of others); this is motivated by the fact that it may not be possible to easily change the data (such as in a biological domain, it may not be possible to change the DNA of a sample). This is mentioned in L176-178 and L183 in the paper.

e) L2 norm & dist function: Note the direction of our condition : we are saying that if the L2 distance between two sets of parameters is small, then their performance is close. Essentially we want to look for modified weights within a C-ball of the original weights. For any reasonable function which does not change abruptly, this condition should hold and give us a set of weights with good accuracy. Your example is that of large distance change but the performance being the same.

Having clarified the condition, there could be other ways of encoding the condition for different models and should be explored by the adversary based on their model architecture.

Dear Reviewer,

Thanks for taking the time to review our paper, we deeply appreciate it.

We are very glad that you find the idea of exploring vulnerabilities in influence functions and our theoretical analysis interesting. Next we address your concerns.

a) Add a discussion: We agree with you that the vulnerabilities are due to flaws in the method itself and would be very happy to add a discussion with the papers you mentioned.

b) Formula on Page 5: As mentioned in L263-269, this formula is meant to capture a baseline for increasing the influence of a target sample without using influence functions. A natural way to do this is to put more importance on the target sample in the loss function, which means the updates will be guided more by the loss on the target sample (eq.5). This can be achieved by either directly reweighing the loss function with different alphas or resampling the target point multiple times in the batch. We found the former to be unstable during training and stuck to the later.

This is the simplest and most intuitive non-influence baseline we could think of. Please let us know if things are still unclear and we will try our best to clarify.

We hope our answers addressed your concerns. Please let us know if there is anything more we could clarify or address. We look forward to hearing from you!

Cheers,

Authors

Thanks for your response. Could you please point to exactly why the real-world scenarios given by us in the response don't convince you? We believe them to be pretty general and realistic.

Dear Reviewer,

Thanks for taking the time to review our paper, we deeply appreciate it.

We are glad that you find the idea of attacking influence functions interesting. Next we address your concerns.

a) Threat model: To begin with, the point is that – there is no one universal threat model or attack which is applicable in all settings. Different circumstances call for different threat models and attacks.

“It seems highly unlikely (especially if the influence calculator can potentially be an adversary) data providers will provide highly sensitive data to this third-party”: There are multiple assumptions in this statement. Firstly, it is assumed that all data is highly sensitive and that providers won’t be sharing this data. In contrast, the idea of data valuation is set in an AI economy where different users are “ready” to sell their data for appropriate compensation and our paper targets such settings. We provide a way this compensation can be manipulated by an adversary and data providers will get a different compensation from what their data is worthy of. We suggest you to read [1,2] to get a better sense of data valuation if interested. Secondly, before our paper, it wasn’t even known that there could be an adversarial setting for influence-based attributions or data valuation at all! This is the point of our paper – to bring to light the potential adversarial actors in this setting. Our paper is an important tale of caution for these very reasons!

Real world scenarios for influence calculator as the attacker: Note that the influence calculator is also compensating data providers. There could be two very natural scenarios for why it is the attacker 1) The influence calculator/model trainer is incentivized financially and wants to save money. Therefore it wants to artificially reduce the compensation it pays to the data providers, which can be done by reducing influence scores. Note that our proposed attacks are symmetric and can be used to both increase or decrease the data value. 2) Some data providers may have colluded with the influence calculator to increase their own compensation (while reducing that of others); this is motivated by the fact that it may not be possible to easily change the data (such as in a biological domain, it may not be possible to change the DNA of a sample). This is mentioned in L176-178 and L183 in the paper.

“another model retraining then”: There is perhaps a slight misunderstanding. The attacked model is not retrained from scratch – few updates are made on top of the original honest model.

b) Linear Models & Simplicity : Note that ours is the first paper which introduces the problem of attacking attributions and proposes attacks with in-depth analysis. We believe the most important contribution of the paper is introducing and motivating the problem of attribution attacks – it is not the methodology or the results or the techniques (though they are very interesting in their own right) – this has opened the doors to future works such as attacking other attribution methods and more complex models. It wasn’t clear how the problem should be formulated or if there is a computationally feasible way of attacking influence functions or if the attacks will succeed even in simpler settings.

The attacks you call simple do not have a 100% attack success rate on even linear models across multiple datasets, which points to the non-triviality of the problem and leads to a very interesting impossibility result (Thm. 1). It is a very interesting theorem in our opinion!

Additionally, we have extra contributions such as an efficient way of backpropagating through inverse-hessian-vector-products – which is of independent technical interest.

We hope our answers addressed your concerns. Please let us know if there is anything more we could clarify or address. We look forward to hearing from you!

Cheers,

Authors

[1]Inflow, outflow, and reciprocity in machine learning. Sundararajan et. al. ICML 2023

[2]Towards efficient data valuation based on the shapley value. Jia et.al. AISTATS 2019 (Influence functions are used as the value estimating functions.)

Dear Reviewer, thanks for your engagement.

The reviewer’s argument is an oversimplification in our opinion, for the following reason.

While it is known that different models can have similar accuracy, it was not known before our paper that a model can be steered to have vastly different and desired influence rankings while maintaining similar accuracy. As an attack paper this is the vulnerability we are exposing. Do you any literature in mind which shows that the model can be maliciously adjusted to produce desired influence rankings?

After this paper it is clear that the benign data providers (though not all have to be benign from our example 2) need some guarantees on the training process. How would this point be made in the absence of our paper?

Cheers, Authors

Dear Reviewer, thanks for your engagement.

The bone of contention here is that you assume if the adversary controls the model training process they will be able to produce desired influence scores (“because models can always be manipulated”,”if they want, they will figure out a way to change influence scores, sure”). You assume that attacking influence functions is a trivial problem — this is where we disagree — our paper is about creating this attack. Why is it obvious that one can produce two models with similar accuracies but desired influence rankings? (this is the literature we were asking for). This is not obvious for even feature-based attributions [1]. The fact that we don’t get 100% attack success rate in our experiments and our impossibility theorem point to the non-triviality of this problem. The claims we make for safeguarding for data providers cannot be made independently of our work just because there exist many attacks in the literature — none of which attack influence functions in particular and whether they can or not is an open question without this work.

We don’t think there is discussion or emphasis in data valuation papers that a trusted model provider is needed. With our paper we make a concrete case that for benign data providers either a trusted model provider or a proof of honest training is required (this is mentioned in our discussion section).

Now answering your question of why hiding training process from the data providers is realistic — We believe this is the usual setting in the real world. Consider a company such as OpenAI or a bank. They buy data from different vendors to build their models*. But do they give the model or show the training process to the data providers? No, the cord is cut at buying the data and the model and its training is proprietary.

Cheers,

Authors

[1] Fairwashing Explanations with Off-Manifold Detergent ICML 2020

*Currently, these deals might be done with negotiation but is subject to change in the future as the field of data valuation progresses.

Dear Reviewer,

Thanks for your engagement.

We believe our threat model is tenable, due to the reasons mentioned in the previous comments. We respect your point of view but would like to end the discussion as of now as we do not see a common ground being achieved.

Cheers,

Authors

Dear Reviewer,

Thanks for taking the time to review our paper, we deeply appreciate it.

We are glad that you find the paper well-written and well-organized, that you appreciate our practical and efficient backpropagation algorithm and our experimental results. Next we address your concerns.

a) Setting of our attack: There seem to be two major concerns you have in this regard. Firstly, why would the influence calculator manipulate the model when it can train it itself. The point is that the model arrived at with naive training may not have the desired influence scores. Therefore the model needs to be steered such that it gives the desired influence rankings, which is what our attack does.

Secondly, what is a realistic scenario where the model training/influence calculator attacks? There could be two scenarios 1) The influence calculator/model trainer is incentivized financially and wants to save money. Therefore it wants to artificially reduce the compensation it pays to the data providers, which can be done by reducing influence scores. Note that our proposed attacks are symmetric and can be used to both increase or decrease the data valuation. 2) Some data providers have colluded with the influence calculator to increase their own compensation (while reducing that of others); this is motivated by the fact that it may not be possible to easily change the data (such as in a biological domain, it may not be possible to change the DNA of a sample).

b) Dist function: The attacker is free to choose whatever method it deems fit for manipulating the model to get desired influence rankings while maintaining accuracy. Along these lines, the attacker can choose any function for implementing dist. We chose the L2 norm for experiments because of its simplicity and ease when using it in Alg. 3 and this choice serves us well in terms of attack success rates!

Note that the attacker wants to maintain accuracy “while getting the desired influence rankings.” It is not clear the method you propose could attain desired influence rankings and is a question for experimentation. Also note that no matter what choice of dist we made, one could always why? So one can also ask why the suggested function is better than the L2 norm? – answers to these questions depend on the particular use-case.

As a side note, in your suggested function, you make an additional implicit assumption that it is allowed to change the model architecture between the original and attacked models (which may not be the case when the influence calculator is commissioned by a client to train a model for a particular architecture or when there are standard architectures leading to optimal performance – so a deviation from the standardarchitecture can be caught by an auditor or when fine-tuning on existing model architectures) and multiplying all parameters by a constant, especially large constants, can be easily caught by an auditor.

c) Complex Models : More complex models are certainly interesting. Based on our intuition of working with influence functions, we do believe that the chances of the attacks working are good since these functions are sensitive to model weights. Regardless, please note that ours is the first paper which introduces the problem of attacking attributions and proposes attacks with in-depth experiments. We believe the most important contribution of the paper is introducing and motivating the problem of attribution attacks – this has opened the doors to future works such as attacking more complex models as you mentioned. It wasn’t clear how the problem should be formulated or if there is a computationally feasible way of attacking influence functions or if the attacks will succeed even in simpler settings. Given this fact, we proceeded with linear models. Even for linear models and multiple datasets, we do not get a 100% attack success rate, which points towards the non-triviality of the problem and leads to a very interesting impossibility result (Thm. 1).

Additionally, we have extra contributions such as an efficient way of backpropagating through inverse-hessian-vector-products – which is of independent technical interest.

We don’t see why the L2 norm would not be a valid choice for dist function in this setting. Regardless, it is up to experimental analysis.

We hope our answers addressed your concerns. Please let us know if there is anything more we could clarify or address. We look forward to hearing from you!

Cheers,

Authors

Dear Reviewer,

Thanks for taking the time to review our paper, we deeply appreciate it.

We are glad that you find the paper easy to follow, clear and also find the idea of studying vulnerabilities in influence functions interesting. Next we address your concerns.

a) Attack setup not real? : There seem to be three major concerns you have in this regard. Firstly, why would the influence calculator manipulate the model when it can train it itself. The point is that the model arrived at with naive training may not have the desired influence scores. Therefore the model needs to be steered such that it gives the desired influence rankings, which is what our attack does.

Secondly, what is a realistic scenario for the model training/influence calculator attack? There could be two scenarios 1) The influence calculator/model trainer is incentivized financially and wants to save money. Therefore it wants to artificially reduce the compensation it pays to the data providers, which can be done by reducing influence scores. Note that our proposed attacks are symmetric and can be used to both increase or decrease the data valuation. 2) Some data providers may have colluded with the influence calculator to increase their own compensation (while reducing that of others); this is motivated by the fact that it may not be possible to easily change the data (such as in a biological domain, it may not be possible to change the DNA of a sample). For attacks by the data provider, please read the next point.

Thirdly, about the plausibility of fairness manipulation. This attack is feasible in scenarios when the methodology we attack is already in use – of using influence functions to weigh samples and train a downstream model as proposed by [1]. Certainly there can exist other ways to alter the fairness of a model, but ours is an influence function paper and we wish to show how different usages of influence functions can be attacked. A simple takeaway from the success of our fairness attack is that perhaps using influence functions to lead to downstream fair models is not right-minded due to the existence of our attack.

Additionally, just want to clarify a possible misunderstanding – we do not have control over training the downstream fair model, we can only train the base model used to find the weights (influence scores) of training samples. Please see Fig.8 in the Appendix for a pictorial representation of the flow.

b) Attacks from the Data provider : To begin with, the point is that – there is no one universal threat model or attack which is applicable in all settings. Different circumstances call for different attacks and threat models.

Now whether attacks can be done at the data providers’ end or not is an interesting question and needs further investigation. Intuitively, data poisoning could be a plausible technique but the most important question is – since the data poisoning should take place prior to model training, what information can the data provider use in the absence of the trained model? Second important question is – since we are talking of relative rankings, how can a data provider raise its own data value wrt others or drop the value for others without access to the data of other providers? Some careful thought needs to go into these questions. Whether or not data poisoning attacks will work for the case of changing influence scores (rather than labels) and at the same time be computationally feasible, can only be confirmed with experimentation and will lead to a different paper altogether. However, we will be happy to add this discussion on data poisoning in the paper.

Having said this, our setting is relevant in cases where data poisoning attacks may not be appropriate – 1) where it is not possible to manipulate even your own data as we mentioned in the paper or 2) not possible to manipulate the data from others due to lack of access. For 1), consider datasets made from DNAs – it is not possible to arbitrarily change data in such settings and also depends on how the DNA (or the data) is collected in terms of source of origin. For 2), a data provider may conduct data poisoning attacks on its own data, but how will it change the data for others to rise relatively in rankings when it doesn’t have access to their data?

c) Overfitting to the test set? : This question is more nuanced than it seems, but there can be simple approaches to prevent overfitting such as stopping early. Now coming to the nuance, since influence scores have to be calculated on test samples, one would always need a test set. Whether your concern is a problem or not depends on where the test set is coming from.

If an external checker/auditor is verifying the influence scores, the influence calculator will show its test set and the calculated influence scores to the checker/auditor since influence scores are tied to the test sets and there is no established consensus or study from our knowledge which measures how much influence scores are sensitive to the test set. If an external party which will use influence scores is supplying the test set to the influence calculator, the influence calculator may overfit. If an external party which will use influence scores keeps a separate test set to check the influence scores, then the influence calculator should use techniques to stop early, while still providing its own test set to the party to show that its calculated influence scores are consistent with the test set it used and prevent getting caught.

d) "Multi-Target Attack" setup: We agree with you, but on the flip side, our greedy attack is the simplest thing one could do (without considering complex training interactions) and yet we get good success rates! This means that our attack is still powerful! Though there is room to see if complicated techniques can do any better.

e) Scale of the data: Increasing the data for which influence scores are to be calculated does make the task harder if we are concerned with top-k/bottom-k rankings. However, our attack can still apply when (1) considering fine-tuning with smaller datasets for specific domains (where the task of data valuation becomes more relevant) or (2) when we consider societal application tasks in finance or healthcare which have smaller datasets, or (3) when we care about altering the absolute value of influence for some data samples.

f) More complex models and other attribution methods: Please note that ours is the first paper which introduces the problem of attacking attributions and proposes attacks with in-depth experiments. We believe the most important contribution of the paper is introducing and motivating the problem of attribution attacks – this has opened the doors to future works such as attacking other attribution methods as you mentioned. It wasn’t clear how the problem should be formulated or if there is a computationally feasible way of attacking influence functions or if the attacks will succeed even in simpler settings. Given this fact, we proceeded with linear models. Even for linear models and across multiple datasets, we do not get a 100% attack success rate, which points towards the non-triviality of the problem and leads to a very interesting impossibility result (Thm. 1).

Additionally, we have extra contributions such as an efficient way of backpropagating through inverse-hessian-vector-products – which is of independent technical interest.

Certainly applicability to more complex models and manipulating other methods (though we believe studying influence functions is also very important given the huge number of use-cases and applications proposed for them) are both interesting questions and have been mentioned by us in the conclusion as future works.

We hope our answers addressed your concerns. Please let us know if there is anything more we could clarify or address. We look forward to hearing from you!

Cheers,

Authors

[1] Achieving fairness at no utility cost via data reweighing with influence. Li et.al. ICML 2022