Principle Counterfactual Fairness

W1. Unfortunately, I believe their problem formulation is flawed by design. The authors only consider the principal strata, i.e. partitions of the data according to a sensitive attribute, for sensitive attributes that causally don't affect the actual outcome. However, why would we ever care about enforcing fairness when that is not the case? Taking the example of the authors: if the task is to estimate whether someone is a good runner, why would we enforce any equality with respect to people with a leg disability, if we already assume that they will not actually be equally good runners? The authors make the same observation in the paper, but fail to discuss the simplest solution: just don't enforce any fairness definition for groups that have no causal effect on the outcomes. In fact, they show in Corollary 1 that their definition of principal counterfactual fairness simplifies to 'standard' counterfactual fairness if we assume that protected attributes indeed have no individual causal effect on all outcomes. However, wouldn't I always either assume the protected attributes have no individual causal effect, or just ignore the group? In fact, making this assumption is a core part of any causal model for fairness.

W2. "This criterion demands that any alterations made to the values of protected attributes do not result in changes to individual decision-making" -> This is highly unclear. Counterfactual fairness demands that changes in protected attributes do not causally result in different decisions, because counterfactual fairness allows changes in the features if the protected attributes change.

W3. Table 1 is also unclear, especially without reading the preliminaries first. How is D(0) different from (D = 1)? Is D(0) the 'do'-operator? I looked it up in Mitchell et al. 2021, and they use D under the do operation of setting A to either 0 or 1. Also, it could be clarified that the definitions of counterfactual parity and cond. counterfactual parity are taken from Kusner et al. 2017, but simply named differently by Mitchell et al. 2021.

W4. The definition of counterfactual equalized odds appears inaccurate. Here, the authors only condition on the product of Y and D = 0, which suggests that the condition is always Y = 1 and D = 0. However, this clearly only implements a form of equalized opportunity. In the actual paper by Mishler et al. 2021, they consider any Y as the condition, in addition to the constraint that D = 0.

W5. Some clarity issues: there are no line numbers, and the authors confuse the spelling of "principle" vs "principal".

Relationship to CF

Overall, I find it challenging to compare the proposed PCF with the original CF in [1]. Is it possible to define principle CF within the SCM framework? More specifically, what would the causal relationships between $A$, $D$, and $Y$ look like in a causal graph?

Experiment

1. My primary concern is about what the authors aim to justify through the empirical study and whether they achieve that goal. Regarding the first question, would it be useful to test previous CF methods to assess if they are indeed too restrictive for PCF? Regarding the second question, in the current draft, the authors show that their post-processing algorithm can improve CF and PCF. How can we be sure that existing fairness or CF methods wouldn’t perform better than the proposed approach?
2. Could the authors provide justification for their choice of dataset? For instance, why not use datasets like Law or UCI Adult, which are more commonly used in CF literature [1][2][3][4]?
3. What is the motivation behind using a causal discovery algorithm and creating subgroups based on its results? This part is unclear to me, particularly since the definition of PCF is not based on an SCM.

Significance

I cannot fully acknowledge the significance of the proposed framework due to the following reasons:

1. As mentioned above, there is confusion about the comparison between CF and PCF.
2. Concerns about the experimental design and its ability to justify the proposed framework’s or method’s contributions.
3. It appears that the authors only provide a necessary condition for an algorithm to satisfy PCF, which relies on the minimum and maximum values of the solution to an optimization problem. I’m uncertain about the effectiveness of this criterion—what is the likelihood that an algorithm could pass this test without truly satisfying PCF? This aspect is not discussed, either theoretically or empirically.

[1] Kusner, M. J., Loftus, J., Russell, C., & Silva, R. (2017). Counterfactual fairness. Advances in neural information processing systems, 30.

[2] Zuo, Z., Khalili, M., & Zhang, X. (2023). Counterfactually fair representation. Advances in Neural Information Processing Systems, 36, 12124-12140.

[3] Kim, H., Shin, S., Jang, J., Song, K., Joo, W., Kang, W., & Moon, I. C. (2021, May). Counterfactual fairness with disentangled causal effect variational autoencoder. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 9, pp. 8128-8136).

[4] Rosenblatt, L., & Witter, R. T. (2023, June). Counterfactual fairness is basically demographic parity. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 12, pp. 14461-14469).

Despite its strengths, the paper has a few limitations. First, while the proposed principal counterfactual fairness criterion is an interesting extension of counterfactual fairness, the contribution feels somewhat incremental. The extension applies primarily to certain narrowly defined scenarios, which may limit its broader applicability and introduce additional challenges in estimation, particularly when defining principal strata for diverse datasets. Second, a notable limitation is the partial identification of principal counterfactual fairness. Since the fairness criterion is only partially identifiable, it is impossible to obtain fully unbiased point estimates, which raises concerns about the potential implications of estimation bias. However, the impact and magnitude of this bias on decision-making outcomes remain unclear, warranting further exploration. Finally, the generalizability of this method to a wide range of applications is uncertain. While the motivating examples provide some context, they are quite specific, and it is not clear how effectively this approach could be applied to varied real-world fairness challenges outside of these examples. Further empirical work and diverse application contexts would help clarify the scope and utility of this method.

-- The theoretical results are not convincing. For example, the independence from the true label in Assumption 1 is strong. Why should the true label be independent of the sensitive attribute? Theorems 2 and 3's results require close estimates of $\hat{\pi}$, $\hat{\mu}$ but the paper does not discuss how these quantities can be estimated using samples.

-- In addition, the experimental results raise additional questions. What is the base value of CF/PCF before post-processing? The paper only shows the increment. Furthermore, if I am not mistaken, only exact fairness is considered. What if we allow approximation? Moreover, what is the effect of imposing fairness on accuracy?

• The contribution is somewhat incremental since it combines two existing fairness definitions.
• Assumptions like ignorability may limit the applications. I think it is more likely this fails in fairness applications specifically. An empirical observation from many social/health sciences is that sensitive attributes are often causally entangled with many/most other relevant variables, so as long as there are any unobserved variables those variables are likely to be confounders (hence ignorability would not hold).
• Overall not important that there are typos of "principle" instead of "principal," but the one in the title of the paper might be important.