Fairness without Harm: An Influence-Guided Active Sampling Approach

We want to thank the reviewer for their positive feedback and comments. We will address individual comments below.

Response to the refinement of validation set: Thank you for raising this concern. For this issue, we did some ablation study in Section 6.3 to explore the impact of the validation set size on our algorithm’s performance, shown in Fig. 2. Our experimental results also demonstrate the superiority of our sampling method over this new baseline.

Response to fairness metric sensitivity of sampling strategy: Thank you for this comment. We want to highlight that our method is general and suitable for all fairness-metric-based fairness losses. It allows for the adjustment of the fairness loss component to suit various fairness metrics as needed.

Response to bias validation set: Thank you for your feedback. In this work, we mildly assume that the validation and test sets come from the same distribution. Consequently, any bias present in the validation set will also be reflected in the test set. Therefore, it is impossible to accurately evaluate fairness on a biased test dataset. Nonetheless, we can discuss some cases where the validation set is biased while the test set is not. Such biases might arise from the small size of the validation set. To illustrate the impact of validation set size, we conducted experiments shown in Fig. 2. If the bias is incurred due to noisy sensitive attributes, we can apply loss correction techniques to these attributes. We welcome the reviewer to provide a practical case of bias for further discussion.

Response to the clean and informative validation set: Thank you for raising this concern. In practice, when dealing with a noisy validation set, we can effectively address the issue by applying a loss correction method to the fairness loss. This adjustment is based on the label transition matrix [1], which helps account for the noise in the labels. Therefore, this practical issue can be heavily alleviated, which can be addressed in future work.

Response to recent studies: Thank you for providing more references. We will update accordingly in the revised version.

Response to the correlation of non-demographic features and demographic information: Thank you for bringing attention to this important aspect. We acknowledge this as an important consideration in data privacy. It is true that some non-demographic features may correlate with sensitive attributes. However, it's crucial to clarify that our method specifically limits its query with the data. We only query the true labels $y$ for a selected subset of unlabeled examples. This approach means that while correlations between non-demographic variables and sensitive attributes may exist in the underlying data, our method itself does not introduce any additional privacy leakage beyond what's inherent in the original dataset. On the other hand, our work’s primary focus is not on addressing privacy concerns related to demographic information. Rather, our main objective is to reduce fairness disparities while maintaining a favorable utility trade-off. We achieve this without explicitly incorporating any additional sensitive information into the process and without directly engaging with the complex privacy implications of demographic data usage.

To address this privacy concern, we can analyze it via differential privacy. Consider a function $\phi(\cdot)$ that maps non-demographic information (features $X$ and label $y$ ) to sensitive attribute $a$. Due to insufficient information, $\phi(\cdot)$ is not deterministic, making it unable to precisely estimate $a$. If a deterministic mapping function existed, it would unavoidably lead to privacy leakage. Therefore, we assume that given the same features and label, the function $\phi(\cdot)$ output might vary. For example, in the Adult dataset, $X$ includes age and credit history, $y$ is the income, and $a$ is the gender. In practice, both men and women have a probability of earning more than 50 K (i.e., $y=1$ ) given the same age, credit history, etc (same feature $X$ ). Suppose $P(\phi(X, y)=a \mid A=a, X, y) \leq 1-\epsilon_0$ and $P\left(\phi(X, y)=a \mid A=a^{\prime}, X, y\right) \geq \epsilon_1$ for all $X, y, a, a^{\prime}$ (where $a \neq a^{\prime}$ ). Here, $A$ represents the sensitive attribute variable, which is unknown to $\phi(\cdot)$. Then, we have $\frac{\mathbb{P}(\phi(X, y)=\tilde{a} \mid A=a, X, y)}{\mathbb{P}\left(\phi(X, y)=\tilde{a} \mid A=a^{\prime}, X, y\right)} \leq \frac{\max \mathbb{P}(\phi(X, y)=\tilde{a} \mid A=a, X, y)}{\min \mathbb{P}\left(\phi(X, y)=\tilde{a} \mid A=a^{\prime}, X, y\right)} \leq \frac{1-\epsilon_0}{\epsilon_1}=e^{\varepsilon}.$ where $\varepsilon=\ln \left(\frac{1-\epsilon_0}{\epsilon_1}\right)$. In practice, if the mapping function is too strong, i.e. $\ln \left(\frac{1-\epsilon_0}{\epsilon_1}\right)$ is too large, we can add additional noise to reduce their informativeness and therefore better protect privacy.

Response to line number and number of datasets: Thank you for pointing out these typos. We will update it in the revised version.

[1] Weak proxies are sufficient and preferable for fairness with missing sensitive attributes, ICML 2023.

We want to thank the reviewer for their positive feedback and comments. We will address individual comments below.

Re to W1: Thank you for raising this concern. We want to clarify that the definitions of well-known fairness metrics, such as DP and EOd, are provided in Section 3 (Preliminaries). These metrics are used for experimental evaluation in Section 6. In this work, we focus solely on active sampling to build a fairer dataset, which is then used to train the model through standard Empirical Risk Minimization (ERM) with Cross-Entropy (CE) loss. Without relying on any additional assumptions about the model or dataset, an intuitive approach is to analyze the loss function. Risk disparity can thus serve as an intermediate-term for theoretical analysis. Proposition 3.1, in Line 162, clearly illustrates the connections between risk disparity and other metrics. Additional theoretical proof can be found in Appendix B.

Re to W2: In practice, when dealing with a noisy validation set, we can effectively address the issue by applying a loss correction method to the fairness loss. This adjustment is based on the label transition matrix [1], which helps account for the noise in the labels. Therefore, this practical issue can be heavily alleviated, which can be addressed in future work.

Re to W3: Thank you for raising this concern. The term "fairness violation" is widely used to quantify the absolute differences based on fairness metrics, such as DP. As outlined in the caption of Table 1, we specifically bold the results that have lower fairness violation meanwhile the accuracy is almost the same as the accuracy obtained by random baseline. For example, in the row for DP on the Celeba-smiling dataset, compared to the random baseline results (0.837, 0.133), we highlight FIS because it achieves a lower DP value (0.084(FIS) < 0.133(random)) while maintaining high accuracy (0.848(FIS) > 0.837(random)). Another baseline, JTT, achieves a lower DP value (0.077(JTT-20) < 0.133(random)), but its accuracy drops significantly (0.698(JTT-20) < 0.837(random)). When applying this criterion to the row for DP on the CelebA dataset, we observe that no results meet our highlighting scheme.

Re to Q1: Please refer to the response to W1.

Re to Q2: Please refer to the response to W2.

Re to Q3: Thank you for your attention. Your understanding is correct. Eq. (5) is compatible with sensitive variables.

Re to Q4 & writing suggestion: Please refer to the response to W3.

[1] Weak proxies are sufficient and preferable for fairness with missing sensitive attributes, ICML 2023.

For your response to W1

I see that the connection with our metrics is mentioned in Line 162. But what I see missing is that, why risk disparity is of interest of this paper? Or what is the context? This is important for readers not for technical reasons but to understand the significance of the metric and thus the proposed method. For example, I could tell the author try to explain it with Figure 1 but still not clear after reading the introduction. Later text jumped into technical details.

For your response to W2 and your response to reviewer cxN7

It's not clear what do authors mean by correction method, nor mentioned in the paper. This is an important piece and should be included at least in the appendix.

For your response to W3

Still I find the presentation of results can be improved.

For your response to Q3

I see that is an advantage of the proposed methods, as many methods could only deal with single sensitive variables. But wouldn't it be more convincing if you use datasets with multiple sensitive variables (both Compas and Adult has only one)?

Overall, I appreciate the rebuttal but I incline to keep my score.

We want to thank the reviewer for their detailed feedback and comments. We will address individual comments below.

Re to W1: We want to clarify that we do not assert that "Those approaches do not strive to achieve uniform accuracy over all groups." Rather, our claim is that these approaches primarily focus on enhancing the worst-affected group performance, and they assess this improvement using accuracy-level metrics. However, this does not necessarily align with popular fairness metrics such as DP. We appreciate you pointing out the need to define the acronym DP clearly. We will ensure this is corrected in the revised version of the paper. Regarding reference [39], as indicated by its title, this paper's starting point is to achieve fairness at no utility cost (predictive utility, synonymous with accuracy). This recent work also motivates us to do active sampling without retraining. Therefore, this paper is within the scope of the fairness-accuracy tradeoff.

Re to W2: Thank you for your feedback. We use the terms "accuracy" and "fairness" to highlight the goals of different components within our model, which helps in distinguishing their roles more clearly. The ‘loss’ component may be inaccurate because both the fairness and accuracy components are based on the loss.

Re to W3: We want to clarify that we do not make any additional assumptions about the iid condition between the train and test data distributions. Our default setting assumes the potential for a distribution shift between two distributions. Our main observations (Lines 297-308) confirm that the proposed method demonstrates robustness under non-iid conditions, effectively addressing potential distribution shifts.

Re to W4: Thanks for your feedback. To illustrate the tradeoff between accuracy and fairness, we consistently present the results in the format: (test_accuracy, fairness_violation). We highlight this form in the captions of tables.

Re to W5: We want to clarify that the JTT weight is a hyperparameter, which can be easily verified because the weight for misclassified examples is input in Algorithm 1. The value of 20 is the same setting as JTT. Here is the original statement presented in the JTT paper: “For JTT, we additionally tune the number of epochs of training the identification model $T$ and the upsampling factor $\lambda_{\text {up }}$. For the final experiments we tune over $\lambda_{\text {up }} \in \\{20,50,100\\}$.”

Re to W6: All baselines including our method are done completely in the same data settings. Due to the space limit, we provide only a brief overview of the three datasets in Section 6.2 (line 331 and line 343). More data information can be found in Appendix E.1 (dataset and parameter settings).

Re to W7: We want to clarify that our main goal is to reach a better accuracy-fairness tradeoff point while not disclosing more sensitive attributes, instead of focusing on less data points. Nevertheless, in Section 6.3, we conduct experiments to illustrate the impact of the solicited data size, shown in Fig. 2.

Re to m-W1: We first use pseudo-labels as in Eq. (5) to find appropriate examples due to the labeling budget, then query the ground-truth labels of the selected examples for training purposes.

Re to m-W3: We want to clarify that the CE loss is the only thing that needs to be minimized during training, rather than fairness loss. The fairness loss is only used to compute the fair influence score. As we highlight in this work, the selected examples should contribute to a decrease of fairness loss by only minimizing the CE loss.

Re to m-W4: There are not many hyperparameters for tuning. In our work, the learning rate and warm-up epoch are fixed.

Re to m-W5: Thank you for pointing out this. The tolerance is used for the final model selection, that is, $\\{\\mathbf{w}\_t \mid \\mathrm{VAL}\_t>\\mathrm{VAL}\_0 - \epsilon \\}$.

Re to m-w2, m-W6 to m-W10: Thank you for pointing out these typos. We will update it in the revised version.

Re to Q1: The <> symbol refers to the vector product.

Re to Q2: We can use an average value for calculation. Mathematically, the only difference between calculating the average gradient and the summation of gradients from the validation set is the normalization factor. However, the absolute values of influence scores are not critical in themselves because these scores are only for ranking purposes.

Re to Q3: In the active sampling setting, the true labels will be inquired for selected data. Due to the large size of the unlabeled set, we first use the proxy labels to select potential “good” examples for inquiring true labels to reduce the labeling budget. Once received the true labels, due to the difference between true and proxy labels, we double-check whether the influence scores of those samples calculated on true labels meet the condition.

Re to Q4: We want to clarify that it is not an assumption but rather a general distribution property. Without making any additional assumptions about the training or testing distributions, this property acts as a direct tool for assessing the differences between the train and test distributions.

Re to Q5: The group gap is for the term $G_k \cdot \operatorname{dist}\left(P_k, P\right) := \operatorname{dist}(\mathcal{A}, \mathcal{B}):=\sum_{i=1}^I\left|p^{(P)}(\pi=i)-p^{(P_k)}(\pi=i)\right|$. Achieving more balance means reducing the distance or disparity in distribution between $P_k$ ​ and $P$, thus allowing for a reduction in the first term.

Re to Q6: Resampling the data to balance data across groups is a regular preprocessing method, which helps prevent overfitting to the majority class. Class imbalance can introduce biases between groups, affecting model performance and fairness. By ensuring a balanced distribution of classes within each group, we can more accurately and fairly evaluate the model's performance.

We want to thank the reviewer for their positive feedback and comments. We will address individual comments below.

Response to weakness: Thank you for raising this good point. We appreciate the feedback and apologize for not emphasizing this aspect in the main text of our paper. We have conducted several experiments, which are detailed in the appendix, to highlight this key point. Specifically, in Appendix E.5, we introduced a new baseline, termed 'random+val', which involves training the model on a dataset created through random sampling combined with a full validation set. As shown in Table 8, including the validation data as part of the training set will degrade the violation of EOp. Our experimental results demonstrate the superiority of our sampling method over this new baseline.

Response to Q1: Thank you for pointing out the concern regarding data leakage. The "original" test dataset has been carefully divided into two independent portions: a new test dataset and a validation set. Therefore, the validation data will not be used for evaluation.

Response to Q2: Equation (4) is correct. In this equation, $\ell(\cdot)$ represents the typical cross-entropy loss, while $\phi(\cdot)$ denotes the fairness loss. Intuitively, we use the inner product of the gradients based on these two types of loss functions for sampling, as shown in Eq. (4). For instance, if the gradient of the fairness loss aligns with that of the accuracy loss for a particular sample, this indicates that incorporating this sample into further training could simultaneously reduce both accuracy loss and fairness loss.