Rethinking Fair Representation Learning for Performance-Sensitive Tasks

We thank the reviewer for their feedback and appreciate the positive comments. Please find our responses to the questions below.

"The paper requires somewhat of a graphical model/ causal background which might be less beginner-friendly. The paper is very dense and requires multiple passes to fully understand the paper. I would recommend adding further intuitions to the paper."

We appreciate that this is a complex topic and our analysis relies on techniques that may not be standard for all researchers. To make our work more self-contained and easy to follow, we will add a brief primer on the background topics in the Appendix, including pointers to further introductory material on graphical modelling and causal inference. We will also go through the paper and clarify the trickiest parts with further explanations and intuitions; if there are any parts in particular you would suggest we focus on, please let us know.

"Do the authors have any comments on the relations between FRL to standard fairness learning schemes that are not representation-based? I would be curious to hear of these problems persist necessarily. "

We agree that these areas are closely connected. Standard FRL methods are designed to enforce demographic parity on downstream tasks; similarly, conditional FRL methods enforce equalised odds. We note in the discussion that our futility result (Proposition 4.8) may be interpreted as an impossibility result between group fairness and minimax fairness, and we point to [1], who explore a similar concept in the literature.

[1] Pfohl SR, Harris N, Nagpal C, Madras D, Mhasawade V, Salaudeen OE, Heller KA, Koyejo S, D'Amour AN. Understanding subgroup performance differences of fair predictors using causal models. NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models 2023.

We thank the reviewer for their review and appreciate the comments on the originality and clarity of our work. We respond to the reviewer’s questions and comments below.

"The causality-based notion in fair ML literature has been well discussed, and there have been numerous causal structures proposed. Can the authors provide a comparative analysis with one or two to further prove the novelty?"

The causal structures of dataset bias we consider are derived by applying the d-separation criterion to find the most fundamental graphs that violate Definition 3.1. They are based on the structures proposed in [1], which also provide practical examples justifying their applicability to real-world settings. These structures may be seen as generalisations of previously studied anticausal bias mechanisms, such as those in [2,3]. Similar structures have also been applied in the robustness and distribution shift literature [4,5].

While the structures themselves were used previously, the techniques we use to analyse them and derive new results are novel. Note also that while we use these structures in our setup and experiments, Proposition 4.8 is not limited to only these structures, as it applies to any structure violating Definition 3.1.

We should also clarify that while our structures of dataset bias are causal, our work focuses primarily on statistical notions of (group) fairness, which are weaker than causal notions of fairness [6,7]. While our work has strong connections to this literature, there are some subtleties to this point, which we also discuss in our response to Reviewer svct (question 1). If you feel that this will improve the paper, we are happy to expand on this discussion and add it to the paper.

[1] Jones C, Castro DC, De Sousa Ribeiro F, Oktay O, McCradden M, Glocker B. A causal perspective on dataset bias in machine learning for medical imaging. Nature Machine Intelligence. 2024 Feb;6(2):138-46.

[2] Makar M, D'Amour A. Fairness and robustness in anti-causal prediction. Transactions on machine learning research. 2023 Feb.

[3] Singh H, Singh R, Mhasawade V, Chunara R. Fairness violations and mitigation under covariate shift. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 2021 Mar 3 (pp. 3-13).

[4] Jiang Y, Veitch V. Invariant and transportable representations for anti-causal domain shifts. Advances in Neural Information Processing Systems. 2022 Dec 6;35:20782-94.

[5] Veitch V, D'Amour A, Yadlowsky S, Eisenstein J. Counterfactual invariance to spurious correlations in text classification. Advances in neural information processing systems. 2021 Dec 6;34:16196-208.

[6] Kusner MJ, Loftus J, Russell C, Silva R. Counterfactual fairness. Advances in neural information processing systems. 2017;30.

[7] Chiappa S. Path-specific counterfactual fairness. Proceedings of the AAAI conference on artificial intelligence 2019 Jul 17 (Vol. 33, No. 01, pp. 7801-7808).

"Limited experiments: it seems the experiments only focus on medical imaging scenarios. The results would be more convincing if datasets from other performance-sensitive areas are considered."

We chose to perform our experiments with medical imaging datasets because the field offers a few interesting and important properties for our analysis. First, the domain is certainly performance-sensitive, so it fits well with our scoping of focusing on performance-sensitive tasks. Second, we often see FRL methods applied to medical imaging benchmarks (e.g. [1,2,3]), demonstrating that this is an active and relevant field of research. Third, the medical imaging datasets we use span multiple diverse modalities, with multiple sensitive attributes per dataset and a wide range of subgroup separability, which was especially useful for our analysis in Sections 5.2 and 5.3.

Our experiments cover five datasets over three separate medical modalities, giving eleven dataset-attribute combinations. We agree that more experiments are always better, and we certainly welcome follow-up work in other domains. However, we feel that this is an appropriate scope for this paper, especially considering that our experiments are only one of the contributions of this work. The diversity of the chosen datasets additionally gives confidence that the conclusions are transferable to other domains and applications.

[1] Zong Y, Yang Y, Hospedales T. MEDFAIR: Benchmarking fairness for medical imaging. arXiv preprint arXiv:2210.01725. 2022 Oct 4.

[2] Pfohl SR, Foryciarz A, Shah NH. An empirical characterization of fair machine learning for clinical risk prediction. Journal of biomedical informatics. 2021 Jan 1;113:103621.

[3] Zhang H, Dullerud N, Roth K, Oakden-Rayner L, Pfohl S, Ghassemi M. Improving the fairness of chest x-ray classifiers. InConference on health, inference, and learning 2022 Apr 6 (pp. 204-233). PMLR.

We thank the reviewer for their positive review and sincerely appreciate the constructive comments and depth of engagement with our work. We respond directly to the questions and comments below, clarifying the key points and suggesting improvements we can make to the paper.

"I’m a little confused by the $X_Z$ and $X_A$ framework, what about features that are impacted by both $Z$ and $A$? It’s not clear if this framework allows for that overlap."

This is an interesting point. In an initial version of this work, we included an interaction feature to explicitly model such situations. However, we realised that this was unnecessary for the proofs in Section 4. This is because the proofs apply to all structures which violate Definition 3.1, of which the three bias mechanisms in Section 3 are simply the most fundamental examples. Furthermore, since the subgraph $A \to X_Z \leftarrow Z$ appears in our formulation of presentation disparities, the interaction case is arguably subsumed by presentation disparities anyway. We will make sure this is clear in the updated version.

Another benefit of this simpler formulation is that it now aligns with the structures proposed by [1], which makes it both easier to justify and more consistent with prior related work.

[1] Jones C, Castro DC, De Sousa Ribeiro F, Oktay O, McCradden M, Glocker B. A causal perspective on dataset bias in machine learning for medical imaging. Nature Machine Intelligence. 2024 Feb;6(2):138-46.

"It would be good at some point to discuss whether any of this applies in an anti-causal setup - the “futility” framing is quite strong and yet I feel as though the large anti-causal class of problems are not really touched on here."

When there is no possibility of label noise ($Y := Z$), notice that our formulation collapses to the familiar anticausal setup ($X \leftarrow Z \to Y$ becomes $X \leftarrow Y$). In this sense, our formulation is a generalisation of anticausal problems, so our results apply to them too. We will add a note to clarify this.

Indeed, while we use our formulation in the problem setup and the experiments, our futility result is slightly more general (as we noted in our response to your other question). One consequence of this is that we can use our theory to derive even more bias mechanisms for which FRL is futile (e.g. $Z \to X \to Y$, with $A$ confounding $Y$ and $X$).

"Quibble with Def 4.5 - technically, I would phrase this as “have an equal amount of information” rather than “do not discard relevant information”. The question of whether information is discarded I think gets more into implementation and I’m not sure how “discarding information” is defined technically."

Thanks for pointing this out, we’re happy to reword this as suggested.

"Quibble with Def 4.4/Lemma 4.6 - I find something a little odd here which I think is around the corner case where ERM representations have no sensitive information. I think the important part of Def 4.4 (the “effectiveness”) of FRL is the equality to 0 - this is whether they are truly “fair” in this definition. However, Lemma 4.6 relies heavily on the first part of the def’n (having strictly less information) - if the ERM reprs have 0 sensitive information then trivially all FRL will violate effectiveness, but this seems wrong as that is not a property of the FRL method, but rather a property of the ERM representations which are given. Anyways, this is not a massive issue with the general point but I do think there needs to be some more careful wording with how the intuition is formalized here (e.g. maybe the I(ERM, A)=0 case needs to be separated out as “trivial”)"

This is an insightful point, and we think we are on the same page about this corner case as you. We thank you for the suggestion to separate this case as trivial; we can add a discussion on this to the Appendix and adapt the main text to signpost it. We think this will help readers who may have similar questions on this part of Definition 4.4.

We thank the reviewer for their thoughtful feedback and are heartened by the comment that reading our work was ‘a joyful experience’.

Below, we address the concerns and comments point-by-point, with suggested changes to the paper to improve it. We are broadly in agreement on many points and are confident that we can address or at least clarify the main concerns; we welcome any further discussion that we hope will give the reviewer confidence to raise their score.

"One major concern is the lack of results in the distributed shift setting. While the negative result in the IID is sound, it would be good to have some results for out-of-distribution, or connects the out-of-distribution fairness to some existed theoretical analysis results in the area of distribution shift."

We agree that theoretical results for this setting would be valuable. While there exists some theoretical work deriving results for closely related methods and settings (e.g. [1,2]), it requires stronger assumptions on the data and/or model than what we wanted to make in our paper. Deriving more general results remains a significant challenge for future work.

This motivates the alternative approach we took of proposing two hypotheses on the empirical validity of FRL under distribution shift and using experiments to test them. Our hypotheses and experimental results are novel and potentially surprising as they are somewhat counterintuitive, which we hope will motivate future research in this area. We highlight this point because we believe the scoping and timely contributions of our work are sufficient to be of interest to the community. While adding more results on this aspect is out of scope, we will highlight this as an important next step in the Discussion section.

[1] Jiang Y, Veitch V. Invariant and transportable representations for anti-causal domain shifts. Advances in Neural Information Processing Systems. 2022 Dec 6;35:20782-94.

[2] Makar M, D'Amour A. Fairness and robustness in anti-causal prediction. Transactions on machine learning research. 2023 Feb.

"Another concern is that the trade-off between fairness and accuracy has been explored by previous methods (Zhao et al., 2022). Why and how this paper is different than previous methods on analyzing the trade-off between fairness and accuracy."

We make distinct and novel contributions that advance the current literature on this topic:

First, by teasing apart three paradigms of fairness analysis in Section 2, we help to expose different assumptions that different studies make about what is considered fair. We feel that this framing will be particularly useful to the community, as it helps to explain why various empirical studies reach seemingly contradictory results. By understanding these distinctions between different paradigms, researchers will be better able to understand and evaluate methods and metrics moving forward.

Second, our causal setup in Section 3 explicitly identifies assumptions about dataset bias and leads directly to our theoretical results in Section 4. We believe that Proposition 4.8 is both the simplest and most general FRL impossibility result, and we credit this simplicity to our proposed causal setup. We think our setup and proof techniques will be useful for researchers aiming to prove other related results in the field.

Third, our consideration of the distribution shift paradigm goes beyond much of the previous literature on fairness-accuracy tradeoffs, which often focuses on tradeoffs in the IID setting (notably [1,2] criticise the typical focus on the IID setting). While we did not include theoretical results for the distribution shift paradigm, our hypotheses and empirical results raise interesting new questions for the field and will help to spark future research directions.

Tradeoffs of fair representation learning have indeed been studied previously, and we believe we give due credit to prior work in this space as thoroughly as possible. An important motivation for our paper was the amount of work that continues to use flawed methods and evaluation practices despite these previous theoretical results. We think that this is, in part, because the fairness literature is vast and complex, and many studies make different implicit assumptions, underscoring the need and timeliness of our work.

[1] Wick M, Tristan JB. Unlocking fairness: a trade-off revisited. Advances in neural information processing systems. 2019;32.

[2] Dutta S, Wei D, Yueksel H, Chen PY, Liu S, Varshney K. Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing. International conference on machine learning 2020 Nov 21 (pp. 2803-2813). PMLR.