Fair Representation Learning with Controllable High Confidence Guarantees via Adversarial Inference

Thank you so much for carefully reviewing our work and appreciating our novelty, the empirical and theoretical results. We address your concern as follows.

Q1: How can this work be extended for other fairness measurements including equal opportunity and equalized odds?

This is an insightful question. Indeed, the proposed method can be applied to Equal Opportunity (EOP) and equalized odds (EOD) with simple modification, but it requires that the outcome labels, Y, for the downstream task of concern are available to compute EOP and/or EOD. We will use EOP as an example. $\Delta_{\text{EOP}}(\tau,\phi) \coloneqq |\Pr(\hat{Y} = 1 |S =1,Y=1) - \Pr(\hat{Y} = 1 | S = 0,Y=1)|$. This is equivalent to $\Delta_{\text{DP}}(\tau,\phi|Y=1)$. The rest follows, except we only consider the subset of the data where $Y=1$ when constructing the high-confidence upper-bound. Similarly, $\Delta_{\text{EOD}}(\tau,\phi) = \max(\Delta_{\text{DP}}(\tau,\phi|Y=0), \Delta_{\text{DP}}(\tau,\phi|Y=1))$ which one can also construct the high-confidence upper-bound using our framework by splitting $\delta$ in half for each of $\Delta_{\text{DP}}(\tau,\phi|Y=0)$ and $\Delta_{\text{DP}}(\tau,\phi|Y=1).$

In the revised manuscript, we will clarify how our approach can be used to constrain EOP and EOD. However, we note that the high-confidence upper bound requires outcome labels, Y, from a specific downstream task, so in this context the guarantee can be applied only to that task, which might not be generalized to arbitrary downstream tasks, as it is the case for the DP (demographic parity). This is a common limitation to existing approaches as well (e.g., [1-2]).

[1] Jovanović et al. Fare: Provably fair representation learning with practical certificates. ICML 23.

[2] Song et al. Learning controllable fair representations. AISTATS 19.

Q2: The IID assumption of the data is a concern to the distribution shift between training and deployment.

We think this is an insightful question. We assume the data are i.i.d. and we agree it could be a concern to some distribution shift. If the test dataset has a different distribution of $(X,S)$, the guarantee that the representation model is fair with high probability may no longer hold. However, the guarantee is robust to label shifts because the label is not necessary when learning the representation model and constructing the high-confidence upper bounds. If the test dataset has a different distribution of label $Y$ or distribution of $Y$ conditioned on $(X,S)$, the guarantee will still hold.

We think that addressing different kinds of distribution shifts of input $(X,S)$ is beyond the scope of our work. Future work can study approaches to construct a high-confidence upper bound with the provided knowledge of the distribution shifts. For example, the shift could be bounded, the shift may occur only to $S$ but not to $X$, the shift may be periodic, etc.

We will include this discussion in our revision.

Q3: The authors proposed solution requires a multi-stage pipeline, which could potential be very computational heavy for large and datasets and complex task. How scalable is such a solution?

Our method requires additional steps for learning the adversary and for performing the fairness test compared to traditional methods. While the training of the adversary and the fairness test introduce some computational overhead, they are very small (around 5% of the overall runtime) compared to the training time of the representation model (95% of the runtime). Here is an example for the Income dataset. We train the representation models with 500 epochs across all baselines. The overall runtime of FRG is 330s, which includes training the adversary, representation model, and the fairness test. Training the adversarial model takes 16s (4.8%), the fairness test takes only 0.06s (0.02%), while training the representation takes about 95% of the runtime. Here is a comparison with the baselines: CFAIR takes 303s, ICVAE takes 1021s, LMIFR takes 336s, LAFTR takes 326s, FARE takes 41s (only one call to fit the decision tree). Thus, we think the computation cost is not a concern. In the revised manuscript, we will include a short paragraph clarifying this.

Q4: How should one consider the thresholds $\epsilon$ and $\delta$?

This is an insightful question. Intuitively, our framework guarantees that any (including adversarial) downstream tasks and models using the representations generated by its learned representation model will have $\Delta_{\text{DP}} <= \epsilon$ with probability at least 1-\delta. How to choose thresholds $\epsilon$ and $\delta$ will vary case by case. If one wants the guarantee to hold with a high probability, one should set a small $\delta$. If one wants to enforce a strict parity constraint, one should set a small $\epsilon$.

Thank you again for reviewing the work, and we will be happy to discuss any further questions you may have.

Thank you so much for carefully reviewing and appreciating our work and raising insightful questions. We address your concern as follows.

Q1: The accuracy of the proposed approach seems too low when the fairness bound is tight (Figure 2).

We think your comment is insightful. On the Adults dataset, when the fairness constraint is strict, e.g., $\epsilon <= 0.08$, the AUCs of our method are around 60-63% which are better than all baselines that satisfy the constraints (ICVAE and FARE). Referring to Appendix Figure 6, we can see that our method achieves the best fairness and accuracy tradeoff. So we agree with you that one can assume there are no practically useful models that can provide such a strict guarantee. In this case, we think it is due to a small $\epsilon$.

Q2: There is little discussion about the assumptions of the proposed model. How realistic these assumptions are and whether they are expected to hold for the datasets considered?

Thanks for the comments. Our main assumptions are provided in Section 7 which include (1) data samples are i.i.d., (2) Student's t-test assumes CLT holds, (3) access to an optimal adversary is available.

Regarding (1), the guarantee could be sensitive to some distributional shifts. If the test dataset has a different distribution of (X,S), the guarantee that the representation model is fair with high probability may no longer hold. However, the guarantee is robust to label shifts because the label is not necessary when learning the representation model and constructing the high-confidence upper bounds. That is, if the test dataset has a different distribution of label Y or distribution of Y conditioned on (X,S), the guarantee will still hold.

Regarding (2), we think this assumption is not very sensitive for most datasets. Our guarantee seems robust for datasets of various sizes (from 41k to 195k). While the t-test assumes normality, it is relatively robust to violations of this assumption, even with small sample sizes, as long as the data are not severely skewed or contain extreme outliers [1].

Regarding (3), we think the choice of the adversary will affect the robustness of the guarantee. One should choose an adversary that achieves high performance on the adversarial task. If a weak model is chosen, it will not approximate the optimal adversary well. In general, we think the best choice of the adversary would vary case by case and be informed by the prediction performance on the adversarial task. While these notes are purely empirical, how to provide a theoretical bound on optimal adversary is still an active research problem and could be studied by future works (this is discussed in the response to Question 1 by Reviewer xgom).

We will provide an extended discussion of each assumption in our revision.

[1] de Winter, J., Using the Student's t-test with extremely small sample sizes, Practical Assessment, Research, and Evaluation 18(1): 10. 2013.

Q3: It is not clear how Theorem 5.2 is used in the paper. What is the benefit compared to $\Delta_{\text{DP}}$?

Thanks for pointing out that this was not clear. Theorem 5.2 is not used to prove that FRG provides the high-confidence fairness guarantees. However, it inspires our choice of the adversarial objective. We define the optimal adversary to be one that maximizes $\Delta_{\text{DP}}$. In Theorem 5.2, we show that maximizing $\Delta_{\text{DP}}$ is equivalent to maximizing $|$Cov$(\hat{Y} , S)|$. Thus, this gives justification that optimizing the adversary with the objective of maximizing $|$Cov$(\hat{Y} , S)|$ can approximate the optimal adversary. Empirically, we show that our design of the adversarial prediction is effective in providing the desired guarantees. We will be sure to clarify this in our revision.

Q4: Are there non-trivial fairness vs. quality tradeoffs with the guarantees provided in the paper?

Thanks for raising this question. The fairness vs utility tradeoff for each dataset is provided in Appendix Figure 6. Our method achieves competitive tradeoffs compared to the baselines. Specifically, on the Adults dataset, FRG achieves the pareto frontier of this tradeoff.

Thank you again for reviewing the work, and we will be happy to discuss any further questions you may have.

Thank you so much for carefully reviewing our work and appreciating our novelty, the empirical and theoretical results. We address your concern as follows.

Q1: How will the relationship between $\Delta_{\text{DP}}$ and $|\text{Cov}(\hat{Y}, S)|$ generalize to equal opportunity or equalized odds?

Thanks for raising this question. We proved in Theorem 5.2 that $\Delta_{\text{DP}}(\tau,\phi) =|\text{Cov}(\hat{Y}, S)|/\text{Var}(S)$. This can be extended to equal opportunity (EOP) and equalized odds (EOD) with simple modification, but it requires that the outcome labels, Y, for the downstream task of concern are available.

EOP difference can be defined as $\Delta_{\text{EOP}}(\tau,\phi) \coloneqq |\Pr(\hat{Y} = 1 |S =1,Y=1) - \Pr(\hat{Y} = 1 | S = 0,Y=1)|$. Thus, this is equivalent to $\Delta_{\text{DP}}(\tau,\phi|Y=1)$. Then, following the same proof, we can conclude that $\Delta_{\text{EOP}}(\tau,\phi) = |\text{Cov}(\hat{Y}, S | Y = 1)|/\text{Var}(S|Y=1)$. That is, the same relationship holds under the additional condition that $Y=1$.

Similarly, EOD difference can be defined as $\Delta_{\text{EOD}}(\tau,\phi) \coloneqq \max(|\Pr(\hat{Y} = 1 |S =1,Y=0) - \Pr(\hat{Y} = 1 | S = 0,Y=0)|, |\Pr(\hat{Y} = 1 |S =1,Y=1) - \Pr(\hat{Y} = 1 | S = 0,Y=1)|)$ which is equivalent to $\max(\Delta_{\text{DP}}(\tau,\phi|Y=0), \Delta_{\text{DP}}(\tau,\phi|Y=1))$. It is also equivalent to $\max(|\text{Cov}(\hat{Y}, S | Y = 0)|/\text{Var}(S|Y=0), |\text{Cov}(\hat{Y}, S | Y = 1)|/\text{Var}(S|Y=1))$.

We will make sure this is thoroughly discussed in our revision.

Q2: The student-t confidence bound relies on CLT; there is no study on how this affects small per-group sample sizes. The CLT approximation could behave poorly.

Thanks for raising this question. This is a reasonable concern when the data size is extremely small. While student-t assumes normality, there is a study that shows that it is relatively robust to violations of this assumption, even with small sample sizes, as long as the data are not severely skewed or contain extreme outliers [1]. In our experiment, we show that the guarantee is robust to various data sizes (41k to 195k). However, if this assumption is violated, our framework is flexible to other choices of statistical tools such as Bootstrap Confidence Intervals, as mentioned in Appendix F for constructing the confidence bounds.

We will extend the discussions on this assumption in our revision.

Q3: The baselines use another dataset for validation, but FRG uses this dataset for both validation and fairness testing. I think that for a fair comparison, the fairness-test dataset should be picked among the training samples.

Thank you for the suggestion. We agree this is a potential weakness in our original setup. We have now completed this new experiment, where the fairness-test set $D_f$ is created from a 10% split of the training data, while keeping all other settings unchanged. The key finding is that our results are robust to this change. The accuracy and fairness outcomes are nearly identical to our original findings (Figure 2-5). We believe that the main reason for this is that the final unseen test set was used identically and fairly across all methods under the old setup. Since the ultimate measure of performance and fairness generalization is on this test set, our central conclusions about FRG's effectiveness hold. We will update the experiment figures accordingly in our revision.

Q4: The pairing part in Sec. 5.2.1. can be explained better. The part where the confidence bound hyperparameter for training, α, can be explained better, is it just the previous $\hat{U}=\alpha U$?

Thanks for the suggestions for improving the clarify of our approach. Regarding the pairing approach, we essentially want to retrieve $m$ pairs of unbiased estimates for each of $\Pr(\hat{Y} = 1|S = 0)$ and $\Pr(\hat{Y} = 1|S = 1)$. They can form $m$ unbiased point estimates of $\Pr(\hat{Y} = 1|S = 0) - \Pr(\hat{Y} = 1|S = 1)$. These point estimates will be used to construct confidence intervals on $\Pr(\hat{Y} = 1|S = 0) - \Pr(\hat{Y} = 1|S = 1)$ by applying statistical tools such as Student’s t-test. We will break down these steps further to make it more digestible.

You are correct that $\hat{U}=\alpha U$. The hyperparameter $\alpha$ is used to inflate the high-confidence upper bound $U$ such that the candidate selection is less likely to overfit to $U$, which may result in more NSF. We will clarify this in our manuscript.

We thank you again for your help in improving our manuscript. We will incorporate your other comments in our revision.

We thank Reviewer xgom for carefully reviewing our work and appreciating the significance of the studied problem and the delivery of the writing and experiment. We provide detailed discussions below to address your concerns.

Q1: The assumption of access to an optimal adversary might not be realistic. Is there theoretical support for the guarantee of obtaining the optimal adversary?

Our theoretical analysis does assume access to an optimal adversary, as pointed out throughout the paper. We address this concern as follows.

First, we empirically validate that our adversary is a sufficiently strong proxy for the optimal one. The adversary is approximated by training an MLP to maximize $|$Cov$(\hat{Y}, S)|$ (Sec. 5.1). This architecture, despite its simplicity, achieves very high AUC, > 98% AUC for each dataset, on the adversarial task when representations are learned without fairness considerations. Our experimental results suggest that the adversary achieved with gradient optimization is sufficient to obtain a valid high-confidence upper bound on the unfairness of arbitrary downstream tasks.

Second, a guarantee for an optimal adversary can be provided by restricting the model and the representation complexities. This approach was taken by FARE[2], which replaces the representation model with the fair decision tree to restrict the representation to be drawn from a discrete distribution with a finite support. While this approach can guarantee access to the "optimal adversary" defined in [2] and provide a theoretically guaranteed upper-bound on $\Delta_{\text{DP}}$, the limitations of FARE will be carried over. In experiment (Sec. 6.2) , we found that FARE can only produce limited variations of representations, so it cannot control the tradeoff between performance and fairness with enough granularity. A lot of times, it is impossible to find a representation model that satisfies the fairness constraints when $\epsilon$ is small, while it is still possible for FRG (Figures 2 and 3). Essentially, this approach violates the controllability, which limits the practicality.

Third, we note that the question of how to provide theoretical guarantees for obtaining or bounding the performance of optimal adversary is a common challenge faced by the field. For example, Feng et al. [7] derived a bound to the accuracy of the optimal Lipschitz-continuous adversary, but the bound is non-tractable in practice. Similar assumptions of access to optimal adversaries have been used extensively by previous works (e.g., [1-6]). Thus, we believe a solution to this challenge merits its own study.

Finally, we also developed an approach that does not require optimal adversary, using a Mutual Information (MI)-based upper-bound on $\Delta_{\text{DP}}$ (Appendix K). This approach gives a theoretically guaranteed upper-bound on $\Delta_{\text{DP}}$ across arbitrary downstream models and tasks that does not assume optimal adversary. However, experiment shows that this upper-bound can be significantly loose (Figure 12) and thus enforcing fairness constraint on this upper-bound is non-practical (Figure 13).

Thank you for raising this important question. We will incorporate a summary of this discussion in the limitations section in our revision.

Q2: Does the choice of adversary design affect the guarantee (the claim in line 189)?

Thank you so much for pointing out that this was not clear. We think the choice of the adversary will affect the robustness of the guarantee. One should choose an adversary that achieves high performance on the adversarial task. If a weak model is chosen, it will not approximate the optimal adversary well. In our implementation, we use an MLP with one hidden layer and tunable hidden sizes as the adversary. We think this architecture is strong enough for the datasets we have tested. We have also verified that this architecture will lead to high prediction performance (> 98% AUC across our three datasets) of the adversarial task if the representations are learned without fairness consideration. We think the optimal adversary is well approximated such that even on the adversarial downstream tasks, we can still guarantee fairness while the baselines cannot. In general, we think the best choice of the adversary would vary case by case and be informed by the prediction performance on the adversarial task. While these notes are purely empirical, how to provide a theoretical bound on optimal adversary is still an active research problem as discussed in Question 1. We will make sure to clarify the choice of the adversary design in our revision.

Q3: Repeated training of adversaries and statistical testing introduces a computational burden. What is the computation cost compared to baselines?

While the training of the adversary and the fairness test introduce some computational overhead, they are very small (around 5% of the overall runtime) compared to the training time of the representation model (95% of the runtime). Here is an example for the Income dataset. We train the representation models with 500 epochs across all baselines. The overall runtime of FRG is 330s, which includes training the adversary, representation model, and the fairness test. Training the adversarial model takes 16s (4.8%), the fairness test takes only 0.06s (0.02%), while training the representation takes about 95% of the runtime. Here is a comparison with the baselines: CFAIR takes 303s, ICVAE takes 1021s, LMIFR takes 336s, LAFTR takes 326s, FARE takes 41s (only one call to fit the decision tree). Thus, we think the computation cost is not a concern. In the revised manuscript, we will include a short paragraph clarifying this.

Q4: The presentation of the experimental results is poor. In Figure 3, the blue line is hard to see.

Thanks for raising this concern. We have increased the marker sizes and used new marker shapes for FRG and they appear much more visible now. We will make sure the final version shows the results as clearly as possible. To address your concern, the blue line achieves very similar results to the orange line, which is also our proposed approach but with supervised loss, thus they overlap. They always satisfy the fairness constraints.

Q5: The choice of datasets is limited, and some of them are outdated. For the adult dataset, you should better use this updated one: "Retiring Adult: New Datasets for Fair Machine Learning"

Thanks for the suggestion. To clarify, we have included the new Retiring Adult dataset you mentioned. We call the new dataset the "Income" dataset (citation [17] in the manuscript) to differentiate from the traditionally used "Adults" dataset. In total, we have included three distinct datasets, each with 2-3 downstream evaluations, which matches most previous works. We will clarify this naming convention in the revision to avoid any confusion.

Q6: Does the proposed method apply to other supervised fairness metrics, such as equal opportunity and equalized odds?

This is an insightful question. Indeed, the proposed method can be applied to Equal Opportunity (EOP) and equalized odds (EOD) with simple modification, but it requires that the outcome labels, Y, for the downstream task of concern are available to compute EOP and/or EOD. We will use EOP as an example. $\Delta_{\text{EOP}}(\tau,\phi) \coloneqq |\Pr(\hat{Y} = 1 |S =1,Y=1) - \Pr(\hat{Y} = 1 | S = 0,Y=1)|$. This is equivalent to $\Delta_{\text{DP}}(\tau,\phi|Y=1)$. The rest follows, except we only consider the subset of the data where $Y=1$ when constructing the high-confidence upper-bound. Similarly, $\Delta_{\text{EOD}}(\tau,\phi) = \max(\Delta_{\text{DP}}(\tau,\phi|Y=0), \Delta_{\text{DP}}(\tau,\phi|Y=1))$ which one can also construct the high-confidence upper-bound using our framework by splitting $\delta$ in half for each of $\Delta_{\text{DP}}(\tau,\phi|Y=0)$ and $\Delta_{\text{DP}}(\tau,\phi|Y=1).$

In the revised manuscript, we will clarify how our approach can be used to constrain EOP and EOD. However, we note that the high-confidence upper bound requires outcome labels, Y, from a specific downstream task, so in this context the guarantee can be applied only to that task, which might not be generalized to arbitrary downstream tasks, as is the case for the DP (demographic parity). This is a common limitation to existing approaches as well (e.g.,[2-3]).

Q7: The details of the model architecture and parameter settings are missing in the paper.

Sorry that the details of the parameter settings are not included in the paper. We use different hyperparameters for different $\epsilon$'s and datasets. We included all of the hyperparameter choices for each of the settings in the config JSON files in the source code provided (as mentioned in Appendix H “Hyperparameter Tuning” referenced in the main text, line 333). The model architecture of our method is discussed in Sec. 5, and the downstream model architecture is discussed in Appendix H. To improve readability, we will also add a summary table of the key hyperparameters choices to the appendix in our revision.

Thank you again for reviewing the work, and we will be happy to discuss any further questions you may have.

[1] Gupta et al. Controllable guarantees for fair outcomes via contrastive information estimation. AAAI 21.

[2] Jovanović et al. Fare: Provably fair representation learning with practical certificates. ICML 23.

[3] Song et al. Learning controllable fair representations. AISTATS 19.

[4] Madras et al. Learning adversarially fair and transferable representation. ICML 18.

[5] Kairouz et al. Generating fair universal representations using adversarial models. IEEE TIFS 22.

[6] Zhao et al. Conditional learning of fair representations. ICLR 20.

[7] Feng et al. Learning fair representations via an adversarial framework. CoRR 19.