Fair Classifiers Without Fair Training: An Influence-Guided Data Sampling Approach

Response to W1: We recognize the parallels between our approach and the existing studies in adversarial training and reliable deep learning where distribution shifts are used to enhance generalization. However, this paper specifically targets using distribution shifts to mitigate fairness disparities without harming the generalization. Then, adversarial training is primarily concerned with improving a model's robustness against adversarial attacks. The focus is on ensuring model stability and performance in the face of intentionally crafted input perturbations. Besides, reliable deep learning focuses on ensuring that these models perform consistently under varying conditions and are resilient to errors, noise, and adversarial attacks. Therefore, this work is distinct in that it specifically aims to tackle fairness issues using distribution shifts, a focus not traditionally central to adversarial training or the broader scope of reliable deep learning.

Response to W2: Our training method is akin to conventional training, involving two stages: a warmup process and an incremental process. Initially, we developed a standard machine learning model without any modifications. In the incremental stage, we implement strategic distribution shifts by assessing the influence of samples on prediction and fairness. Samples that exert a negative influence on both prediction and fairness loss — indicating an enhancement in both fairness and prediction — are then solicited for further training. Apologies for any confusion regarding the term “influential”. It's analogous to the influence function concept, where we select unlabeled samples based on their influence on prediction (Lemma 4.1) and fairness (Lemma 4.2), hence the term 'influential examples'. More specifically, in Line 6 of the FIS Algorithm, we select those unlabeled samples with negative influences on prediction and fairness. By doing so, we aim to reduce fairness disparity without compromising accuracy. Our use of "influential" differs from previous uses in literature, which often focus on identifying misclassified examples from training sets by using influence functions [1,2]. In contrast, we only assess the influence of prediction and fairness on the validation sets $Q_v$ for unlabeled samples by applying first-order gradient estimation, following the traditional training process.

Response to W3: We respectfully disagree with the comment that our paper places undue emphasis on a particular aspect. Previous research often presupposes the availability of demographic details like gender and race, known as sensitive attributes, for training fair models, particularly in in-processing methods.

When sensitive attributes are missing, some works tend to find some proxy models to generate proxy attributes [3]. However, in our method, we need neither true sensitive attributes nor their proxies in the training. There are some others mentioned by Reviewer B8Mi and Reviewer wM5N, including DRO to improve fairness without sensitive attributes. Although they also evaluate the worst-group performance in the context of fairness, their approach differs as they do not strive to equalize the loss across all groups. In our revised manuscript, we intend to elaborate on other studies that conduct training without using sensitive attributes. Additionally, we respectfully appreciate any relevant references you may provide.

Response to W4: Taking into account the feedback from other reviewers, we plan to provide a more comprehensive overview in the 'Related Work' section. This will encompass topics such as distributionally robust optimization, the accuracy-fairness tradeoff, distribution shifts, and adversarial training.

Response to Q1: Adversarial training works on existing training data while predicting adversarial behaviors at deployment/test time, while ours (strategic distribution shifts) actively solicits new samples to augment the training dataset.

Response to Q2: Motivated by the key observation of theoretical findings, in section 4, we propose an influence-guided sampling strategy that actively samples new data to supplement the training set such as mitigating fairness disparity. Besides, we are confident that the overall content in Section 4 is new. In subsection 4.1.1, we restate the foundational scenarios and further introduce the extra validation set $Q_v$ and the unlabeled set U. Following this, subsection 4.1.2 delves into the methodologies used to determine the influence on predictions and the fairness component. These concepts are integral to the FIS algorithm we describe in Section 4.2.

[1] Just train twice: Improving group robustness without training group information, ICML 2021.

[2] Achieving Fairness at No Utility Cost via Data Reweighing with Influence, ICML 2022.

[3] Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributes, ICML 2023.

Response to Q3: The use of distribution shift is inspired by theoretical insights from Lemma 3.1 and Theorem 3.2. These findings shed light on the reasons behind the tradeoff between accuracy and fairness, guiding us to manage this tradeoff by controlling the distribution shift's negative impact on accuracy via generalization error. Without specific information on the train/test distribution, determining the precise distribution shift required is challenging. However, our primary goal is to leverage this distribution shift to minimize the group gap without harming model accuracy.

Our approach differs in its fundamental focus and methodology. While adversarial training is geared towards enhancing a model's robustness against intentionally crafted perturbations, and influence function-informed training aims to understand and mitigate the impact of specific training examples, our method primarily addresses fairness against potential data shifts by adding more examples. We focus on generalization across varying data distributions rather than explicitly countering adversarial examples or pinpointing influential data points.

Response to Q4: Sorry for missing this part of the information. We will add captions for tables in the revised version.

Response to W1: Work [1] primarily concentrates on improving the worst-group accuracy of neural networks in the presence of spurious features, which seems irrelevant to the fairness topic. Work [2] mainly focuses on the use of distributional robust optimization (DRO) to upweight the training set to improve group robustness. Similar to DRO [2], [3] proposes an optimization approach, Adversarially Reweighted Learning, to improve worst-case performance over unobserved protected groups. In reality, [4] is an expansion of the work presented in [2], with a primary focus on the fine-tuning of hyper-parameters. This extension specifically addresses the sensitivity of various hyper-parameters, such as the size of the validation set. [5] exploits these estimated subclasses by training a new model to optimize worst-case performance over all estimated subclasses using group distributionally robust optimization (GDRO). These works explicitly focus on improving the worst-group loss. Although they also evaluate the worst-group performance in the context of fairness, their approach differs as they do not strive to equalize the loss across all groups. Besides, in these studies, accuracy and worst-case accuracy are used to showcase the efficacy of DRO. Essentially, they equate fairness with uniform accuracy across groups, implying a model is considered fair if it demonstrates equal accuracies for all groups. However, this specific definition of fairness is somewhat restrictive and does not align with more conventional fairness concepts like Demographic Parity (DP) or Equalized Odds (EOD).

Response to W2: While fair active learning frameworks may seem similar to our approach, there are crucial distinctions. For instance, work [6] focuses on sampling additional unlabeled data points based on a multi-objective optimization for misclassification error (Shannon entropy) and fairness regularization (demographics parity). In contrast, [6] still requires knowledge of the sensitive attributes of unlabeled samples, while our method does not necessitate such information. Practically, without this information, it's difficult to assess fairness disparity (e.g., using demographics parity) and determine if a model is fair. To overcome this, we resort to a validation set that includes sensitive attributes, providing a reliable measure of fairness disparity.

Response to W3: Those in-processing fairness training algorithms typically require sensitive attribute information to develop regularization terms. For instance, MinDiff [7] employs various regularization methods to reduce bias so as that obtain an accuracy-fairness tradeoff. Consequently, comparing our approach with in-processing algorithms is unnecessary and not fair, as these algorithms are not applicable in scenarios where sensitive attribute information in the training data is unavailable.

Response to W4: The fine-tuning of hyper-parameters, such as the size of the validation set, is crucial due to the use of proxies and the implementation of Distributionally Robust Optimization (DRO) [2, 4]. More specifically, [2] employs misclassified examples from a standard Empirical Risk Minimization (ERM) model to represent minority sub-groups, while [4] generates pseudo-sensitive attribute labels on validation data. In contrast, our approach utilizes a straightforward, traditional gradient-based sampling method, simplifying implementation and reducing the complexity of hyper-parameter fine-tuning. In our experiments, it is important to note that the hyperparameters specific to each dataset are the same regardless of the sensitive attributes or the binary classification targets. In our experimental setup, it's crucial to point out that the hyperparameters specific to each dataset remain constant. To address your concern, we will conduct experiments on these hyper-parameters, including the impact of solicitation budgets. These additional results will be added to the revision before the rebuttal deadline.

Response to W5: Thank you for your feedback. Regarding your concern, we plan to conduct a sensitivity analysis to examine how the labeling budget impacts both accuracy and fairness.

Response to W1: We appreciate your observation and agree that there are similarities in methods, yet our approach significantly diverges in its specifics. First, unlike the influence function method, we employ a first-order gradient to select the samples, verified in Lemma 4.1 and Lemma 4.2. We would like to emphasize that our method focuses on soliciting additional samples from an external unlabeled dataset (consistent with a fair active learning framework) while [1] reweights the existing and fixed training dataset. Benefiting from this, our approach is more suitable for larger numbers of unlabeled data because it allows for straightforward incremental training on a basic model, rather than the complete retraining required by [1].

Response to W2: Due to the lack of specific information about the train/test distribution and sensitive attributes in our scenario, crafting a perfectly tailored algorithm that directly links to our theorems to control distribution shifts is challenging. To address this, we've introduced the heuristic sampling method FIS, focusing on utilizing gradient descent to effectuate an appropriate distribution shift. While our work aligns with certain domain adaptation theories, it primarily investigates the underlying causes of the accuracy-fairness tradeoff, extending beyond just domain adaptation. By controlling the adverse effects of distribution shifts, our goal is to strike a more favorable tradeoff between accuracy and fairness.

Response to W3: The choice of first-order estimation, rather than considering the second-order information, is due to the nature of training deep neural networks with non-convex objectives subject to fairness constraints. In such scenarios, the model often does not converge to a local minimum where the first-order gradient can be treated as zero, thus the first-order term will dominate over the second-order term. In recent literature [R1], the authors have shown that the first-order estimation per sample is nearly accurate under fairness constraints. In the uploaded revision, we added Figure 5 to compare the estimated influence with the actual influence. We observe that while some of the examples are away from the diagonal line (which indicates the estimation is inaccurate), the estimated influence for most of the data samples are very close to their actual influence values.

Response to W4: Apologies for not providing sufficient information in the table captions. We will revise the table captions to include more details and upload the revised version. In those tables, we report the values of accuracy and corresponding three fairness metrics in the format: (test accuracy, fairness metric). Although the tradeoff might not always be apparent in practice, the results shown in the tables demonstrate that we can reduce fairness disparity without sacrificing model accuracy.

[R1] Understanding instance-level impact of fairness constraints, ICML 2022.

Thanks for your comment! We apologize for any inconvenience. We have uploaded the latest paper. In revision, more discussion has been added in Section 2 (Related Work). Then, Figure 4 compares the estimated impact with the actual impact. Please always feel free to let us know if we could assist with understanding the paper or addressing any concerns.

Response to W1-1: We would like to clarify that we do not work in the classical regime of fairness-accuracy tradeoff. By properly collecting new data from an unlabeled dataset, we can improve both generalization and fairness at the same time which can not be achieved by working on a fixed and static training dataset that naturally incurs such tradeoff. Another notable distinction between our work and previous studies on the accuracy-fairness tradeoff lies in the handling of sensitive attributes of training data. For instance, [1] achieves this tradeoff using fairness regularization (in-processing techniques), which necessitates sensitive attribute information. In contrast, our paper's key contribution is the elimination of this information, thus offering a more practical and applicable framework in scenarios where such information is unavailable.

Response to W1-2: Thanks for the suggestions. Overall, our approach significantly differs from other studies that concentrate on fairness disparity arising from train-test distribution shifts. Commonly, such research necessitates extra assumptions to build theoretical connections between features and attributes, like causal graphs [2], correlation shifts [3], and demographic shifts [4]. In contrast, our approach refrains from making further assumptions about the characteristics of distribution shifts. Instead, we directly utilize sampling methods to construct an appropriate distribution shift, aiming to achieve a balanced tradeoff between accuracy and fairness. More specifically, work [2] uses complex causal graphs to hypothesize about distribution shifts between training and testing datasets. Following this, Then, [3] introduces and analyzes the correlation shifts between train and test distribution, serving to boost existing fair in-processing methods. However, to our knowledge, fair in-processing methods necessarily require information on sensitive attributes during training, whereas our approach does not require this information. [4] control unfairness of a model by making the demographic shifts assumptions, where the marginal distribution of the data remains the same conditional on the subgroup but the subgroup distribution can change.

Response to W2: [5] proposes an approach based on distributionally robust optimization (DRO), which minimizes the worst-case risk over all distributions close to the empirical distribution. Then, [6] also proposes an optimization approach, Adversarially Reweighted Learning, to improve worst-case performance over unobserved protected groups, indicating the main goal is to improve the worst-group loss. Although these two works [5, 6] also evaluate the worst-group performance in the context of fairness, their approach differs as they do not strive to equalize the loss across all groups. Besides, in these studies, accuracy and worst-case accuracy are used to showcase the efficacy of DRO. Essentially, they equate fairness with uniform accuracy across groups, implying a model is considered fair if it demonstrates equal accuracies for all groups. However, this specific definition of fairness is somewhat restrictive and does not align with more conventional fairness concepts like Demographic Parity (DP) or Equalized Odds (EOD). In response to your concern, we are considering using one of DROs (e.g., [5]) as a baseline for our ongoing experiments. These additional results will be added to the revision before the rebuttal deadline.

Response to W6: As far as we are aware, there don't seem to be fitting baselines for comparison in our case. We are receptive to any suitable baseline suggestions you might have. Although some studies train models without using sensitive attributes [4-8], they may not serve as direct baselines for our work. This is because these works explicitly focus on improving the worst-group loss. Although they also evaluate the worst-group performance in the context of fairness, their approach differs as they do not strive to equalize the loss across all groups. Besides, in these studies, accuracy and worst-case accuracy are used to showcase the efficacy of DRO. Essentially, they equate fairness with uniform accuracy across groups, implying a model is considered fair if it demonstrates equal accuracies for all groups. However, this specific definition of fairness is somewhat restrictive and does not align with more conventional fairness concepts like Demographic Parity (DP) or Equalized Odds (EOD). In addressing your concern, we plan to use one of DROs [4, 7, 8] as a baseline in our current experiments. We will incorporate these results into the revised manuscript before the deadline for rebuttals.

Response to Weakness (minors): Sorry for these typos. We will carefully revise the references in the revision.

Response to Question 1: This seems to be a straightforward understanding that theorem 3.2 tells you that if there is no distribution shift between the training and testing set, the generalization error would be small and bounded. While avoiding distribution shifts contributes to a small error, this also maintains a large fairness disparity, evident as a group gap in Theorem 3.2. This observation has been validated by numerous existing works that focus on the accuracy-fairness tradeoff. In this paper, we are trying to build connections between train and test data by using our actively enhanced data (including the initial labeled training data and new solicited data), following a train data $\rightarrow$ solicited data $\rightarrow$ test data sequence. If we can shift the distribution of training data towards that of test data, we can reduce the fairness (group) gap. It's important to note that we cannot control the fairness gap from train to test data, which is determined by the nature of the problem. Our control lies in strategically shifting the training data distribution by adding more samples, with the hope that this will reduce the inherent fairness gap between train data and test data.

Response to Question 2: Your understanding is correct. We define $x$ as features in the first paragraph of Section 3.1(Preliminaries).

[4] Just train twice: Improving group robustness without training group information, ICML 2021.

[5] Achieving Fairness at No Utility Cost via Data Reweighing with Influence, ICML'22

[6] Fairness without demographics through adversarially reweighted learning, NeurIPS 2020.

[7] Hyper-parameter Tuning for Fair Classification without Sensitive Attribute Access, arXiv 2023.

[8] No subclass left behind: Fine-grained robustness in coarse-grained classification problems, NeurIPS 2020.

Response to W1: We apologize for any raised confusion. In our work, we represent the training dataset using the notation P and the test dataset with the notation Q, without assuming specific characteristics for these sets. This implies that P and Q could either be independent and identically distributed (iid) or exhibit a distribution shift. Conventionally, it's a standard approach to train a model on the training set P and evaluate its effectiveness on the test set Q.

Response to W2: We would like to clarify that the assumption of L-lipschitz smoothness (Assumption 3.1) and bounded gradients (Assumption 3.2) of the empirical risk are common and basic in the theoretical analysis of stochastic gradient descent (SGD) and related optimization methods in machine learning [1, 2, 3]. For your reference, these two assumptions are similar to Assumption 1 and Assumption 4 presented in [1], respectively. More specifically, the L-Lipschitz property for the empirical risk is inferred from the L-Lipschitz property of the loss function because the empirical risk $R_P(w)$ is defined as the average of the loss function over the training dataset. On the other hand, gradient clipping is a widely applied trick to avoid exploding gradients in deep learning, thus implying that the bounded gradient is a decent assumption.

Response to W3: We respectfully disagree with the comment that the assumption of the validation dataset is too strong. This assumption is widely recognized and has been presented in numerous existing works [4-8]. In our setting, the distribution of the data remains unknown to us because merely accessing a small validation dataset for testing purposes does not fully reveal the data distribution. Besides, we can analyze this assumption in two scenarios: whether the train and test distributions are independent and identically distributed (iid) or not. If they are i.i.d., the validation dataset can be drawn from the training dataset. However, in cases of distribution shifts between train and test data, we cannot ascertain the influence of solicited data samples without any explicit knowledge of the test data. Therefore, we utilize this reasonable assumption to gain some insight into the test dataset or its distribution.

Response to W4: We would like to clarify that the extra computation cost is comparable to the cost of traditional model training. The main extra computation cost in FIS (Algorithm 1) is the cost of model gradients. Let $p$ denote the number of model parameters, then the cost for computing the gradients is $O(p)$ per sample. Specifically, in each round that we need sampling, we need to calculate three parts of gradients: the gradients of $N_U$ unlabeled instances, the average gradient of $N_v$ validation instances wrt accuracy loss, and the gradient of $N_v$ validation instances wrt fairness loss. Note in general, $N_v \ll N_U$. In practical implementation, to speed up the calculation of gradients over $N_U$ instances, we randomly sample 0.2～0.5 percent of the unlabeled dataset in each sampling batch. Additionally, we can increase $r$ to save computation costs. In our experiments, we usually have 10～20 sampling rounds. On a single GPU, running one experiment for CelebA requires about 4 hours.

Response to W5: In our work, we do not make specific assumptions on the initial training dataset. In our experiments, we randomly select a subset of examples as the initial labeled training set. In response to your concern, we are conducting experiments to show the impact of solicitation budgets. These additional results will be added to the revision before the rebuttal deadline.

[1] On the Convergence of FedAvg on Non-IID Data, ICLR 2020.

[2] Fair resource allocation in federated learning, ICLR 2019

[3] SCAFFOLD: Stochastic Controlled Averaging for Federated Learning, ICML 2020.