Towards Poisoning Fair Representations

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

On the necessity of working at representation level.

Attacking mutual information at representation level aligns better with fair representation learning (FRL) and is more universal than attacking $I(Y(X), a)$. For example, a bank may want to obtain a representation for each user that can be used to determine their eligibility for existing and upcoming financial products such as credit cards without concerning about fairness again [1]. Here each financial product is associated with a (unique) label, and determining the eligibility entails a classification task. In this case, there are two challenges for delivering a $Y(X)$-based attack. First, one has to determine and justify which classifier $Y(\cdot)$ to use and why consider the fairness of this specific classification task. Second, for any upcoming financial product, its label $Y$ does not exist and one cannot obtain classifier $Y(X)$ (one needs $Y$ to train such a classifier), let along attacking it. In contrast, a representation-level attack can overcome the two challenges in a single shot. As discussed in section 2.2, by maximizing $I(z, a)$, any $I(Y(X), a)$ will be maximized so long as the fairness concern exists. This implies launching attack on all labels $Y$ simultaneously, including the ones where classifies $Y(X)$ cannot be trained.

Issues in the discussion related to other MI proxies.

We appreciate the detailed explanation and valuable feedback from the reviewer. We've revised the remark 2.2 in the paper as follows:

Maximizing I(z,a) is a general framework to poison FRL and admits other proxies such as sliced mutual information (Chen et al., 2022), kernel canonical correlation analysis (Akaho, 2007), and non-parametric dependence measures (Szekely et al., 2007). In this work, we use FLD because of its conceptual simplicity, interpretability, and good empirical performance. As one may recognize, when $p(z | a = 1)$ and $p(z | a = 0)$ are Gaussian with equal variance, FLD is optimal (Hamsici & Martinez, 2008) whose BCE loss attains the tight lower bound of $I(z, a)$ up to constant $H(a)$. In this case, our method provably optimize the lower bound of I(z, a) whereas other proxies whereas other proxies may not due to the lack of direct connections to mutual information. While the Gaussianity may not hold in general, FLD score as a measure of data separability is still valid (Fisher, 1936), and we verify its efficacy for our goal in Appendix D.2, where we show that FLD score is highly informative for the empirical minimal BCE loss of a logistic regression.

Slightly limited evaluation.

As pointed out by the reviewer, we focus on simple tabular data because many fair representation learning methods focus on these data [1, 2, 3, 4]. Moreover, previous work on attacking classical fair machine learning also only focused on tabular data [5, 6, 7], and we follow this convention.

To better evaluate the effectiveness of our method, we added experiments on two more benchmark datasets COMPAS and Drug Consumption. Results have been updated in Appendix D.8. We reported decrease of BCE losses in Figure 15, increase of DP violations in Figure 16, and accuracy of predicting $y$ from $z$ in Figure 17. Again, on two new datasets our attacks successfully outperformed AA baselines to a large extent. Observing these good performance, we will study the vulnerability of bias mitigation methods for CV and NLP tasks as our future work.

[1] Zhao et al. Conditional learning of fair representations, 2019.

[2] Moyer et al. Invariant representations without adversarial training, 2018.

[3] Creager et al. Flexibly Fair Representation Learning by Disentanglement, 2018.

[4] Reddy et al. Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics, 2021.

[5] Solans et al. Poisoning attacks on algorithmic fairness, 2020.

[6] Mehrabi et al. Exacerbating algorithmic bias through fairness attacks, 2021.

[7] Chang et al. On Adversarial Bias and the Robustness of Fair Machine Learning, 2020.

I thank the authors for their effective rebuttal. Most of my concerns have been addressed. I therefore update the score to 8 in response to your efforts. The discussion on why to attack at a representation level is indeed insightful and instructive.

My final suggestions are to (a) highlight the motivation for attacking at a representation level in the revised manuscript; (b) improve the presentation by e.g. avoiding inline figures and moving all figures to the top/bottom of a page; and (c) remind the reader the potential risk of violating the Gaussianity assumption in high-dimensional cases (and refer them to the ablation study). I would be very interested to see when will FLD begin to fail (and even if it fails for some d, it is still a good contribution).

Thank you for your good rebuttal. Question 1 is important to conduct this work since I see other reviewers also raise similar concerns, and authors conduct extra experiments on this concern. I think this work's contribution is improved after rebuttal so I rasie my rating for contribution evaluation. Hope other reviewers can retrospect the rebuttal content : )

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

Authors use their own defined vanilla metric, and lack related fairness-aware metrics like Equality odds (EO).

In section 2.2 we analyzed how our attack can deteriorate the well-known fairness-aware metric demographic parity (DP). In experiments we verified that our attack successfully exacerbated DP violations of predicting label $y$ from representation $z$ learned by four victims on four datasets (Adult, German, and new COMPAS, Drug Consumption). Due to page length these results were deferred to Appendix D.3 (Adult and German datasets) and Appendix D.8 (COMPAS and Drug Consumption datasets) respectively.

Authors are encouraged to conduct more experiments on more datasets like COMPAS and Drug Consumption.

Thanks for your suggestions, we conducted experiments on COMPAS and Drug Consumption datasets and reported results in Appendix D.8 of our updated paper. We reported decrease of BCE losses in Figure 15, increase of DP violations in Figure 16, and accuracy of predicting $y$ from $z$ in Figure 17. Again, on two new datasets our attacks successfully outperformed AA baselines to a large extent.

Personally, I reckon authors are encouraged to conduct experiments on deeper NN (I think simple MLP is not that DEEP to be called "DNN"), though the datasets are relatively simple.

To see how our attacks perform on deeper and larger NNs, we tested how fair representations learned by CFAIR and CFAIR-EO can be attacked when the NNs are larger. Experiments were conducted on the largest Adult dataset. We increased depths of encoder to 3 hidden layer and adversarial classifier to 5 hidden layer. We increased the dimension of representations to 128 and set all hidden sizes to 60. Due to time constraint, we did not tune the model architectures to obtain the best clean performance. Training epochs were increase from 50 to 100, and all other hyper-parameters such as batch size and learning rates were unchanged. As shown in the table below, our attacks succeeded in all settings while three baselines suffered from several failures.

Finally, we would like to mention that the choices of model architecture and training settings can have crucial influence on the performance with and without attack, so these results are just for illustration. Moreover, motivated by the promising results, we plan to extend our attack to fair machine learning on large-scale image and text datasets.

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

One of the problem of introducing elastic penalty is how to choose properly the two penalty parameters...Would it make significantly difference if we simply choose the L1 norm penalty instead?

We chose the penalty parameters following the convention in previous work [1] and the choices were briefly selected from a relatively large range, therefore, we believe that the attack performance can benefit from a fine-grained search for better parameter values.

Regarding the advantage of using elastic-net penalty than using $L_1$ norm penalty only, conceptually, elastic-net penalty is able to choose a group of correlated features whereas using $L_1$ or $L_2$ penalty only cannot [2]. Empirically, to check the effect of using $L_1$ norm penalty only, we tried the same choices of $\lambda_1$ as in the main experiment and compare the corresponding attack performance and perturbation magnitude. Due to time constraint we only conducted experiments on Adult dataset and reported results in Appendix D.5 of the updated paper.

In short, we note that comparing with elastic-net penalty, $L_1$ norm penalty usually resulted in perturbations with larger $L_1$ and $L_2$ norms but the attack performance was hardly improved, especially when small- to intermediate- level penalty parameter values were used.

[1] Chen et al. Ead: elastic-net attacks to deep neural networks via adversarial examples, 2018.

[2] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net, 2005.

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

Could the authors clarify their threat model more clearly?

As the first work towards conducting poisoning attack against fair representation learning (FRL) methods, we followed previous works on attacking classical fair machine learning such as [1, 2, 3] and assumed a strong attacker that has full knowledge to the victim model: including its architecture, training details, and parameters. This type of attack is known as white-box attack and has been widely used to benchmark the vulnerability (robustness) of a model [1, 2, 3, 4] and is necessary to develop a black-box attack in general (in the sense that an optimization-based attack is often conducted by attacking a collection of white-box surrogated victim models) [5, 6]. Therefore, we believe that this requirement does not undermine our contribution on understanding how vulnerable FRL methods are. Nonetheless, we agree with the reviewer that a black-box attack would be more practical and we will delve into developing stronger and more transferable attack towards FRL methods in our future work.

[1] Solans et al. Poisoning attacks on algorithmic fairness, 2020.

[2] Mehrabi et al. Exacerbating algorithmic bias through fairness attacks, 2021.

[3] Chang et al. On Adversarial Bias and the Robustness of Fair Machine Learning, 2020.

[4] Koh et al. Stronger data poisoning attacks break data sanitization defense, 2022.

[5] Huang et al. MetaPoison: Practical General-purpose Clean-label Data Poisoning, 2021.

[6] Geiping et al. Witches’ brew: Industrial scale data poisoning via gradient matching, 2021.

Could the authors give more insight to explain why their attack cannot be mitigated by fair representation learning (FRL)?

Thanks for your question. We plan to explore this direction in our next step towards understanding the vulnerability of existing fair representation learning (FRL) method. Here we give two possible causes.

First, fair machine learning methods often assume that the underlying data distribution is unchanged. When a distribution shift appears, many of these methods showed degraded performance [1, 2]. We presume that FRL methods suffer from a similar vulnerability as well, i.e., their training is not robust enough against distribution shift. Consequently, a data poisoning attack, which injects carefully-crafted poisoning samples to the training data, can be seen as imposing an adversarial distribution shift to the empirical training distribution. This may deteriorate the fairness performance of FRL methods on the validation (targeted) data that follows the original distribution.

Second, as mutual information is difficult to estimate and optimize, existing FRL methods also rely on approximate solutions that require training of some auxiliary models such as a sensitive feature predictor to conduct adversarial training or a deep neural network-based mutual information estimator. In practice, these auxiliary models may also encounter unstable training and result in sub-optimal fairness results, and the performance degradation can be exacerbated by the data poisoning attack.

[1] An et al. Transferring Fairness under Distribution Shifts via Fair Consistency Regularization, 2022.

[2] Jiang et al. Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach, 2023.

I think that the authors should clarify this and position better their paper and contributions with respect to other existing works.

The main contributions of our work lie in two-fold. First, we provide a new MI-based attack goal to degrade fairness of high-dimensional representations $z$. Specifically, we maximize $I(z, a)$ where $a$ is the sensitive feature. In contrast, previous attacks mostly focused on scalar predictions on $\hat{y} = h(z)$ and seeks to maximize $I(h(z), a)$ (loosely speaking) which is much easier to evaluate than $I(z, a)$ as $z$ is high-dimensional. As discussed in section 2.2, the benefit of using this new MI-based attack goal is that by maximizing $I(z, a)$, for any classifier $h$, $I(h(z), a)$ is free to grow (so DP violation increases [1]) so long as the fairness concern exists. Notably, this condition holds even if ground truth $y$ is unobserved so that previous attack goals cannot be defined.

Second, we provide a theoretical analysis on the required poisoning samples to launch a gradient-matching based attack. This analysis also sheds light on defending against the proposed attack method and can be of independent interest to other gradient-matching based attackers.

To avoid possible confusion, we've updated this statement as "We propose the first data poisoning attack that directly targets on fairness of high-dimensional representations as shown in Figure 1."

[1] Zemel et al. Learning Fair Representations, 2013.

The experimental evaluation is not convincing: In the experiments the authors just reported results evaluating the BCE loss but did not consider any of the existing metrics for measuring the fairness gap.

We reported violation of demographic parity (DP), one of the most widely-used fairness notion, from the four victim models trained on Adult and German datasets in Appendix D.3. In the revised paper we further added corresponding results on COMPAS and Drug Consumption datasets in Appendix D.8 of updated paper. In all cases, our attacks successfully increased DP violations, clearly showing its effectiveness.

For some of these attacks, the authors can use the same outer objective and use the same approximation for solving the bilevel optimization problem.

As analyzed in the responses above, we believe that previous outer objectives may not be good choices for our purpose and they indeed fail to apply in certain cases. For example, in [1] the outer objective is a weighted average of classification accuracy and relaxed group fairness gap. However, ICVAE-US does not involve any classification in its training, therefore, these two terms are undefined.

[1] Van et al. Poisoning Attacks on Fair Machine Learning, 2021.

The authors say: “Heuristics such as label flipping (Mehrabi et al., 2021) lack a direct connection to the attack goal, thereby having no success guarantee and often performing unsatisfactorily.” I think this is not true. Although more limited on the attacker’s capabilities, smart manipulation of the labels can lead to successful attacks.

Thanks for your thought. We meant to express that label flipping-based attacks do not directly optimize the attack goal, and the goodness of these manipulations can be hard to quantify. As a result, one may not provide theoretical analysis on the minimal number of poisoning samples as we did for the gradient-matching based attack. However, we agree that smart label flipping attack can lead to successful attacks in practice, and we have updated this statement as "Heuristics such as label flipping (Mehrabi et al., 2021) do not directly optimize the attack goal, thereby often requiring great effort to design good manipulations to success."

We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.

The paper lacks a clear threat model: it is unclear whether the attacker’s objective is useful for compromising algorithms with and without mechanisms for mitigating the fairness gap. On the other side, it is unclear what is the attacker’s objective and the relation of the attack strategy with respect to the model’s performance.

In this work we proposed a new objective to attack fair representation learning (FRL) models by maximizing the mutual information (MI) between high-dimensional representations $z$ learned by the victim and the sensitive attribute $a$. This attacking objective aligns well with the well-known MI-based fairness goals used in FRL as detailed in section 2.1. Moreover, this objective jeopardizes the well-known fairness metric demographic parity (DP) of any classifiers predicting label $y$ from representation $z$ as discussed in section 2.2, and is empirically verified in our experiments covering four FRL models trained on four well-studied benchmark datasets. We further added accuracy of predicting label $y$ from representation $z$, and found that our attacks had minimal impact on the accuracy.

It is unclear why this strategy is better compared to other attacks already proposed in the research literature

Existing works, including the listed one, focused on attacking fairness in classification tasks, i.e., the fairness is directly defined and attacked based on scalar predictions. We instead seek to degrade the fairness of high-dimensional representations directly, which is a much more challenging task and requires a new formulation as discussed in section 1.
Notably, FRL methods do not always need access to label $y$ during the training time, see ICVAE-US for an example. For such victims, all existing poisoning attack formulation fall short to apply while our formulation is able to handle them.

There is not mention to “Subpopulation Poisoning Attacks” by Jagielski et al., which are also very relevant to this work and proposes more scalable alternatives for crafting poisoning attacks targeting fairness. & The authors used a gradient matching strategy for approximating the solution of the bilevel optimization problem. However, Jagielski et al. (“Subpopulation Poisoning Attacks”) shows that other strategies can be more efficient for this. I think this aspect requires further analysis.

Thanks for bringing to us this work. After reading through it, we agree that subpopulation attack is a direction that deserves more exploration in the future study on deriving stronger attack against FRL methods, and we have updated the paper to discuss this accordingly. At the same time, we found subpopulation attack less relevant to the current main focus of our paper. First, a subpopulation attack, which identifies and attacks the most vulnerable subgroups of data, fills the gap between the availability and targeted attack. While in this work, we focus on an availability-like attack, where the attacker seeks to degrade fairness guarantee for as many samples as possible (on the whole validation set). Second, as mentioned by the reviewer, subpopulation attack provides more scalable alternatives to craft poisoning samples, while the main focus of this paper is to formulate an attack goal towards fairness of high-dimensional representations and to propose a tractable proxy that can be used as an objective for poisoning samples crafting.

Equation (2) relies on strong assumptions about the distribution (Gaussian distribution and continuous variables) of the different subpopulations. How is this a good proxy for approximating the original problem? How does this compare with existing attacks (as the ones mentioned before)?

Fair representations are often real-valued [1, 2, 3, 4] and thereof we believe that the "continuous variable" assumption naturally holds in general fair representation learning scenarios.

In terms of the Gaussian assumption, as discussed in remark 2.2, when this assumption holds, Eq (2) is not only a proxy, but also an optimal solution. When this assumption does not hold, using FLD (Eq (2)) as a data separability measure is still valid [5] and we believe that it is straightforward to use data separability as a measure of how difficult it is to classify them and as a proxy for the optimal BCE loss one can achieve on this classification task (which also depicts the classification difficulty). Notably, this BCE loss is always a lower bound of the mutual information in the original problem. In comparison, to our best knowledge, there is no analysis on how existing attack goals (as the ones mentioned before) relate to the mutual information between high-dimensional representations $z$ and sensitive feature $a$.

Empirically, Figure 4 in Appendix D.2 verified that FLD score acted pretty well in approximating the optimal BCE loss to classify different classes of fair representations learned by four different fair representation learning methods. In short, in this work, we use FLD because of its conceptual simplicity, interpretability, and good empirical performance.

[1] Zhao et al. Conditional learning of fair representations, 2019.

[2] Moyer et al. Invariant representations without adversarial training, 2018.

[3] Creager et al. Flexibly Fair Representation Learning by Disentanglement, 2018.

[4] Reddy et al. Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics, 2021.

[5] Fisher. The use of multiple measurements in taxonomic problems, 1936.

How assumption 2 works in the current threat model for the attack? The authors say: “Before the attack, the victim is well trained.” What does this mean?

A gradient matching-based attack relies on a trained victim model [1]. To simplify the theoretical analysis, assumption 2 states that this trained model is converged to a local stationary point of the lower-level objective before being attacked. This assumption is reasonable because the lower-level objective is the one that the model is trained on. Moreover, it allows us to bound the influence of clean gradient from clean training data, and to derive a lower bound for the minimal number of poisoning samples. This bound is valid in the sense that when the assumption is violated, typically a larger amount of poisoning samples is needed.

The rational behind is as follows. When training the victim on poisoned data, each mini-batch consists of two types of samples: poisoning samples will push the victim model to optimize the upper-level objective (attack goal), whereas clean sample will counteract with this goal and instead force the victim to optimize the lower-level objective. By assuming that the victim has converged with respect to the lower-level objective, how clean samples counteracts with poisoning samples will be minimized: they only add randomness to the poisoning gradients. This allows us to derive the bound in a similar way to stochastic gradient descent.

[1] Geiping et al. Witches’ brew: Industrial scale data poisoning via gradient matching, 2020.

Also, for the theoretical analysis: Why do the authors think that assumption 3 is reasonable?

Similar to assumption 2, assumption 3 allows us to simplify the derivation of the lower bound. This assumption assumes that all poisoning samples in hand are well-crafted. This assumption can be achieved by conducing the gradient matching on poisoning samples one by one. As gradient matching is a standard optimization problem, modern optimizers can solved it readily. In addition, assumption 3 does not require all poisoning samples to match the upper-level gradient exactly. Instead, each of them only needs to form an unbiased estimation. Notably, when the assumption is violated, typically a larger amount of poisoning samples is needed, and the derived lower bound is still valid.