On the Fairness ROAD: Robust Optimization for Adversarial Debiasing

1. The motivation of this paper is that global fairness methods usually cannot guarantee a consistent fairness value on split subregions, shown as Fig. 1. However, such observations are based on fine-grained partition (e.g., age categories) of training data and each subregion might be with a relatively small size. In this case, the disparity tends to happen. For example, DI might be very large if a bin only includes two samples. How do we confirm that biases do widely exist and then precisely define “local fairness”?

While our definition of local fairness is given in Eq.(3) as means of expectations for a specific family of $q$ distributions, we acknowledge that measuring fairness from finite datasets may face difficulties. Given a considered partition of $X$, the presence of small subgroups can easily result in large DI values, and hence, it is essential to define a threshold to properly demonstrate the existence of the local fairness problem. This threshold is to be defined depending on the use case, and represents the minimum group size for a measure of local fairness to be deemed significant.

In our work, we do use such thresholds when measuring local fairness: we highlight the existence of the problem (Figure 1) by considering a minimum of 50 individuals in each represented age category (Adult and COMPAS). This threshold allowed us to observe significant disparity between sensitive groups, and conclude on the existence of the local unfairness problem. The local fairness results presented in Figure 2 and Figure 5 follow the same methodology: subgroups of minimum 50 individuals for the COMPAS dataset, and 20 for the Law and German datasets (details for all subgroups are given in Appendix A.5.3).

Besides, to make sure that the problem of local fairness we observe is not an artifact of overfitting a threshold, we have conducted multiple experiments to test the robustness of our observations: in Figure 5, Figure 9, and Figure 10, we test the behavior of models trained once when this threshold varies. These experiments show that no matter the value of the threshold chosen (among a reasonable range), local bias problems could be observed.

Finally, we would like to restate that these thresholds are only needed at evaluation time, to measure local fairness. During the training, no threshold is needed, as the locality of the attention is controlled by the temperature $\tau$ hyperparameter.

2 .Following question 1, I think it might be necessary to use some prior for split subregions. For example, in DRO, they introduced a lower bound of the subgroup ratio. With accuracy as a utility, BPF demonstrated that very small group would lead to uniform classifier. But from sec 3.2, as Softmax weights can be regarded as applying a Shannon entropy regularization. It would be better if the authors can point out such behind insights. Blind Pareto Fairness and Subgroup Robustness, ICML 2021.

Thanks to the reviewer for this very insightful remark. Since considering worst-case distributions, mentionned works [1] and [2] have indeed connections with our work, that we could discuss.

The DRO approach [1], as [3], assumes P as an unknown mixture of distributions and attempt to minimize the risk of the worst components of that mixture. Using the dual form of the problem, they consider an upperbound of the risk that depends on the marginal of the smallest component of the mixture. To avoid degenerating through trivial solutions that outputs uniform predictions, they indeed set a minimal threshold for the size of that smallest component they could focus on. This threshold is shown to be related to the maximal divergence allowed between their worst-case distribution $q$ and the training one $p$.

Please note that their setting is different than ours: their goal is to train models that perform uniformly well across all partitions of the population, while ours is to train a model that is uniformly fair (regarding a sentitive attribute) accross all subregions of the feature space, which is quite different. In particular, they usually consider convex losses and contraints to build their models, while we envison more complex architectures designed through neural networks. However, setting a prior on the least populated group for which we can ensure fairness can be seen equivalent as tuning our hyper-parameter $\tau$, that controls the Shannon entropy of our divergence ratio $r$ (defined as a maximum entropy distribution using softmax weights).

We thank again the reviewer for their comments, that led us to include several components of the above discussion in the paper in sections 2.2, 2.3 and 3.

[1] John Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses for latent covariate mixtures. Operations Research, 71(2):649–664, 2023.

(1) The paper does not directly address limitations. For instance, is there a computational scalability issue with this technique compared to other more well known group fairness methods?

Thank you for your comment. In terms of computational scalability, the training process of our method is relatively stable, and the networks do not require many neurons (3 layers with no more than 64 neurons per layer). Our study in the appendix A.4.1, titled “Does the complexity of the two adversarial networks have an impact on the quality of the results?” shows that for network r, a simpler linear prediction such as logistic regression is quite efficient (please note that due to the exponential parameterization, an exponential activation is applied to it). We believe that our method strikes a balance between complexity and the potential improvements in local fairness it offers. For limitations, see our answer to Question 2 below.

(2) The reference to applied fairness research could be strengthened. There are many related works that are omitted. In particular, work on fairness and adversarial learning.

We appreciate the suggestion to strengthen the references to applied fairness research and related works. We have included the following references in the manuscript: Applied fairness:

[5] Dissecting racial bias in an algorithm used to manage the health of populations, Obermeyer 2019

[6] Achieving Fairness through Adversarial Learning: An Application to Recidivism Prediction, Wadsworth et al. 2018

[7] A Fair Pricing Model via Adversarial Learning, Grari et al. 2022

Some missing related works on adversarial learning for fairness:

[8] Achieving Fairness through Adversarial Learning: An Application to Recidivism Prediction, Wadsworth et al. 2018

[9] Learning to Pivot with Adversarial Networks, Louppe et al. 2017

(1) Is there a reason why the first paragraph of the introduction omits citations in statements like "models...have been shown to unintentionally perpetuate existing biases and discriminations"?

Thank you for your comment. In our revised manuscript, we have included more relevant literature to better situate our work within the broader context of fairness research. These include for instance:

[10] Machine Bias, Angwin et al. 2016 (Propublica Compas study)

[11] Weapons of Math Destruction, Cathy O’Neil, 2016

(2) What do you see as the main use case for this method? What are the limitations you anticipate other than sensitive attribute not being available?

Usecases

Our contribution generally aims towards a better generalization of fairness for classification tasks. In this context, we believe it should be considered as an alternative in all problems and usecases already tackled by the field of algorithmic fairness: healthcare, credit obtention, etc. More particularly, several works in applied fairness have observed and discussed problems of “fairness generalization” or “local unfairness”. These usecases would thus represent an especially interesting area to explore. Some of them include:

[12] focus on mitigating bias in a task of harmful tweet detection using several datasets. The sensitive feature is whether the tweet content is aligned with racial dialect (e.g. African American English). However, they observe that despite generally managing to enforce bias, the models showed poor generalization capabilities across datasets. In such situation, ROAD might help ensuring better fairness generalization.

[13] observed that, in a context of car insurance pricing, disparities among groups persist in some geographical regions even after mitigating bias on the policyholder’s residence using adversarial methods. Our approach can be a powerful tool in addressing such local fairness issues.

[12] Differential Treatment: Mitigating Racial Dialect Bias in Harmful Tweet Detection, Ball-Burack et al. 2021

[13] A Fair Pricing Model via Adversarial Learning [Grari et al. 2022]

(1) Complexity of problem for formulation: The authors seek to impose a robust version of the demographic parity constraint in (3), by introducing a maximization over an uncertainty set in the constraint. Instead of directly seeking to solve this problem, they formulate an equivalent problem with an additional argmin on an adversary's reconstruction loss (4). Could authors elaborate why they chose to work with (4) instead of the simpler form in (3)? It appears that the trick of applying instance-level weighting function could be applied as well to solving (3).

Thank you for your question. We chose to work with (4) because it allows us to introduce an adversary’s reconstruction loss and make the problem differentiable, which greatly helps optimization, specifically when working with neural classifiers. Directly solving (3) would be intractable as the function $\hat{f}$ is non-differentiable after thresholding ($\hat{f}_{w_f}(x)=\mathbb{1}_{f_{w_f}(x)>0.5}$), making it unsuitable for gradient-based optimization. Our approach is inspired by the fairness reconstruction in adversarial settings, as demonstrated by Zhang et al. (2018), which also propose to enforce fairness by mitigating the ability of an adversary to reconstruct the sensitive attribute S from the probability f(X). Theoretical justification of this kind of adversarial approach for fairness is given in the second paragraph of Appendix A.2.2.

(2) Assumptions on $q$: Could the authors formally state the assumptions on the class of perturbed distributions $q$, preferably early on in the paper (e.g. Sec 2).

In section 2, we give an overview of the problem, by giving a general formulation (eq 3) and then reviewing some related works that attempt to address fairness with DRO approaches. In section 2.3, we review different choices for the considered uncertainty set ${\cal Q}$, which has a strong impact on the behavior of the methods in practice. We indeed see from eq (3) that if ${\cal Q}$ is too large or far from $p_{test}$ (the target distribution for the classifier), the process might lead too overconstrain the prediction, with the consideration of fairness for irealistic groups, which results in poor accuracy for the target classifier. Thus, for distribution shifts purposes followed by related works, the main requirement for ${\cal Q}$ is that it contains $p_{test}$, and is tight around it. Assuming that $p_{test}$ is close to $p$, this is done for instance by a KL constraint w.r.t. $p$.

For our local fairness objective, we also want that the discrepancies of $q$ w.r.t. $p$ are smooth in the feature space, so that the fairness constraint does not increase mitigation on specific disconnected individuals, but rather on local areas of the space, which would be somehow ignored in classical global fairness approaches.

Following the reviewer's recommendation, we added this last sentence in section 2.3 to strengthen the definition of our requirements regarding ${\cal Q}$. However, from our point of view, discussions about our validity constraints and the implied assumptions can difficultly be discussed here, for the reasons detailed below.

My understanding is that the group priors are required to be the same for both $q$ and $p$, i.e. $q(s) = p(s)$.

The constraint of the initial problem from Eq.(3) compares expectations over prediction outputs for both sensitive groups, without any dependency on a given $q(s)$. $q$ only acts conditonnally on the binary sensitive value $s$ (the max is on posteriors $q(x|s)$).

In its transposition in an adversarial formulation that is more tractable (eq.5), each sample is considered via an individual loss. It could be tempting to act on $q(s)$ to induce more powerful perturbations, but we show theoretically in Appendix A2.2 that it would induce a shift of the optimum, supported by empirical results in Appendix A2.3, by allocating higher $r$ weights to the most populated group (we added a new figure 6 in section A.2.3), to emphasize that most of the fairness effort is supported by the most populated group in this setting). In order to avoid over-mitigate one subpopulation of the sensitive compared to the other one, we thus proposed to add an additional constraint to ensure that the marginal $q(s)$ equals the observed prior $p(s)$ through the definition of $r$.

I thank the authors for the detailed response and clarifications, and for the revisions to the paper.

I have a few follow-up questions:

• Rationale for assuming equal priors: Thanks for the clarification. I am still not entirely convinced about the choice of uncertainty set. If "s" is a sensitive attribute that defines gender (for simplicity, male and female), am I correct that you consider perturbations to the data distribution that maintain the same overall proportion of male and females? If so, why are we only considering perturbations of this form? Wouldn't it natural to also expect data drifts that change the overall proportion of male and females?

• Differentiability of (3): I understand that formulation (4) allows for differentiability of the objective. Couldn't the same be achieved by introducing a smooth surrogate objective for the non-differentiable indicators? Much of the initial works on group fairness employ some form of surrogate approximations to handle non-differentiable constraints (e.g. Zafar et al. (2015), Cotter et al. (2019)).

P.S. Regarding my earlier point about the condition q(s) = p(s) imposing a constraint of equal priors across all sub-regions, let me be a bit more clear. When you compute the worst-case loss over all distributions q for which q(s) = p(s), you also include within this set distributions q that have uniform density over samples in a particular age group, and zero density on other samples. Such distributions define a sub-region of your data, but also additionally satisfy the condition q(s) = p(s) within the sub-region.

[2] Duchi et al. 2020 Distributionally Robust Losses Against Mixture Covariate Shifts. ROAD is a heuristic approximation to the objective

[3] Michel et al. 2021 Modeling the Second Player in Distributionally Robust Optimization

[4] Michel et al. 2022 Distributionally Robust Models with Parametric Likelihood Ratios

Fig 4 (Left Fig): Are these unnormalized values? Isn't weight, i.e., $r$ positive and less than 1?

Normalization is on the mean, not the sum: we impose $\mathbb{E}_p[r(x,s)]=1$ to ensure that $q$ is a distribution that respects $\int q(x,s)=1$. As we implement this constraint with an unbiased estimator based on training samples (i.e., $\frac{1}{n} \sum r(x_i,s_i)=1$), we get that $r$ values sum to $n$, not $1$. This can be seen for instance in our implementation of BROAD where the partition function is divided by $n$.

Parameterization of weight network: If we use BROAD for EO, it will use the adversary's loss, which uses labels $y_i$. Thus $r_i$ is a function of $x_i$, $s_i$ and $w_i$. Should the weight network (in ROAD) also use $y_i$ to predict $r_i$?

Yes, we agree with the reviewer and confirm that in order to optimize for EO, $r$ needs to be a function of $x$, $s$ and $y$; and that the weight network uses $y$ to predict $r$. We have provided more details about this in the appendix of the paper (please see A.10: "ADAPTING ROAD TO EQUALIZED ODDS").

EO, validity constraint: Would the validity constraint change for EO? That is, wuld it require normalization over both $s_i$ and $y_i$, or only $s_i$ as explained in the paper?

Thank you for your question. Following similar reasoning as for our validity constraint in the case of DP (section A2.2), we have indeed also considered adapting the validity constraint for EO by conditioning it on $y$. This effectively means imposing the following constraints: $\forall s_i, \forall y_i, \mathbb{E}_{x | s=s_i, y=y_i}r(x,s,y)=1$ However, depending on the dataset, conditioning the batches on both $s$ and $y$ may lead to very small subsamples, i.e. non-significant expectation estimates. As a result, the obtained models were sometimes less robust, and this lead us to not pursuing this direction any further. We have added a discussion on this in Section A.7 with an experiment comparing the two constraints (Figure 13).

We thank the reviewer for the valuable comments. We address the various questions/concerns below.

Baseline: In fig 3 and 4, some of the baselines do not span all ranges of fairness. Why do we see this behavior? Are the baselines tuned correctly?

We assume that by “fig 3 and 4”, the reviewer instead meant fig 2 and 3. In these figures, some curves indeed do not span all ranges of fairness. This is due to the fact that some methods do not offer sufficient control, depsite an extensive sweeping of their hyper-parameter domains via greed search, that would enable to reach points in some areas of the support. For fig 2, this is accentuated by a filter that removes points that do not respect a minimal threshold of global fairness (as we are interested in inspecting local fairness for globally fair models). For fig 3, this is emphasized by distribution drifts: while for in-distribution test set (in the leftmost curves), every method allows a wide span of the global fairness support, results for other plots are impacted by drifts that make pareto fronts reduced on few points dominating every others for some methods.

How does DRO affect global fairness compared to other baselines? Fig 3 only reports local fairness in the form of the worst DI. However, the global fairness results are not reported.

In fact, that figure reports results of local fairness for a high requirement for global fairness (global DI < 0.05). We see that our approach achieves this global fairness level (while some approaches such as fairLR on COMPAS have no point in the plot since never satisfying this condition), with competitive levels of accuracy for any local fairness score.

While extending this framework to EO measure is straightforward, results with EO and local fairness are not reported. Similarly, DI and distribution shift results would strengthen confidence in the method.

You are right. We launched experiments to complement this. We have added a new section in appendix "A.6 Additional results on the Local Equalized-Odds criterion" (page 20). You may find these results in Figure 12 in the current version of the paper. The local unfairness is here measured using worst Disparate Mistreatment Rate (worst-1-DMR), that we defined by adding worst-1-FPR and worst-1-FNR.

ROAD vs BROAD: It is quite unintuitive that BROAD, an exact solution, performs poorly than ROAD, which uses a neural network to predict weights. The authors argued that ROAD is better because of the smoothing effects. Could it be that what is easy and hard for adversaries may change rapidly (or adversaries' output may be noisy) due to noisy batch gradients, and using an NN in ROAD may just be smoothing that out? In that case, would it make sense to use an EMA-like estimate with BROAD?

Thank you for this insightful question regarding the difference in performance between ROAD and BROAD.

The problems that may hinder BROAD performance go beyond smoothing. In the DRO literature, it is known, and deeply discussed (see [1, 2, 3, 4] for instance), that the non-parametric solution (such as BROAD) may lead to suboptimal performance compared to parametric ones. This may come from several causes, such as the fact it is essentially solving a problem that is fundamentally too pessimistic: i.e., the uncertainty set $\mathcal{Q}$ may include distributions that are too difficult to learn, and not necessarily representative of real-world constraints. Thanks for the suggestion of using an EMA-like estimate with BROAD, which could alleviate learning unstability issues, but might still be limited by the use of individual weights: in BROAD, weights of training samples are only interlinked via the outputs from the classifier, hence at the risk of conflicting with our notion of local fairness (which is defined on the feature space).

On the other hand, the parametric, neural network-based, approach (such as ROAD), “has much less (only parametric) freedom to shift the test distribution compared to the adversary that uses non-parametric weight” (sic) [1]. The lipschitzness of neural networks can add additional implicit locality smoothness assumptions, thus helping define distributions $q$ as subregions of the feature space. In classical DRO, this leads to models that are robust to more meaningful distribution shift. In our setting of DRO for local fairness, this also helps focusing on local fairness with groups formed with similar individuals. As discussed in our paper, the network architecture defines the level of local smoothness of considered groups. And a network of infinite capacity that completes training would have, in theory, the same behavior as BROAD.

[1] Hu, Weihua, et al. “Does distributionally robust supervised learning give robust classifiers?.” International Conference on Machine Learning. PMLR, 2018. [2] Duchi et al. 2020 Distributionally Robust Losses Against Mixture Covariate Shifts. ROAD is a heuristic approximation to the objective