Counterfactual Fairness from Partially DAGs: A General Min-Max Optimization Framework

(A) The motivation of introducing IDA algorithm seems unclear

To enumerate possible adjustment variable sets, the authors employ the existing IDA algorithm, which aims to find adjustment variables from the siblings of sensitive feature $A$. However, I could not understand why it is sufficient to focus on the siblings. I understand that it is easier to determine whether a variable is parent of $A$ if it is a sibling of $A$ (because of the property of v-structure). However, I could not understand why all possible confounder can be enumerated with the IDA algorithm.

• Could you elaborate why employing IDA algorithm is sufficient? I believe that if it fails to output possible confounders, the resulting propensity score model cannot eliminate the confounding bias.

(B) Clarity issues

• I could not understand why the authors present the formula of $P( \hat{Y}_{A \leftarrow a} = y)$ in Section 3.3. Is it used in the proposed method? If so, it is computationally demanding since there are so many possible values of $pa(A) = z$ if $z$ is high-dimensional.

• Please clearly state that prediction $\hat{Y}$ is regarded as a node (an endogenous variable) in the causal graph in Section 2.2. Otherwise, it is difficult to understand the problem setting. Is observed outcome $Y$ also a node in the causal graph, right?

I have two major concerns, which keep me from accepting the paper in its current state.

1. Identifiability of counterfactuals. The paper claims to achieve (point) identification of counterfactual predictions. However, I believe point identification is only possible here because, in Proposition 3.4., the authors do not condition on the descendants of the sensitive attribute $A$. This is in strong contrast to the main claim of the paper to use all observed variables for counterfactual fairness, not just the non-descendants. Furthermore, it is inconsistent with the proposed counterfactual estimator on page 6., in which I believe the authors do include descendants of $A$. In general, estimating counterfactual predictions would require (at least implicitly) estimating the counterfactual descendants of $A$ (i.e., causal queries of the form $\mathbb{P}(M_{A \to a^\prime} \mid A = a, M = m)$, where $M$ are descendants of $A$. The identifiability theory of such causal queries is well-developed, and counterfactual (point) identification is impossible without imposing additional assumptions (Plecko and Bareinboim, 2022). Looking forward, I believe the authors have three options to improve the paper: (i) Imposing strong parametric assumptions on the data-generating process that enable counterfactual identification (and I think it is necessary that these assumptions are spelled out explicitly). (ii) Using a partial-identification approach, as done by Wu et al. (2019) for discrete variables. However, Melnychuk et al. (2023) showed that, for continuous variables, not even partial counterfactual identification is possible without imposing stronger assumptions. (iii) Choosing a weaker/less individualized (but identifiable) notion of fairness, such as interventional fairness (Kilbertus et al. 2017).

2. Min-max approach for achieving counterfactual fairness. The authors motivate their approach with the toy example in Figure 1. However, if I understand correctly, the proposed approach would coincide here with the FAIR method, as the worst-case DAG would include both $X_1$ and $X_2$ as descendants. More generally: Why does the min-max approach proposed in the paper not coincide with the FAIR method that excludes all (possible) descendents of $A$? I.e., why does the worst-case DAG not always include all (possible) descendants as actual descendants? I could imagine that this would induce certain conditional independences that would violate the learned Markov equivalence class. But I think it is important that the authors provide some intuition here, either via a theoretical result or another toy example, that shows that the proposed max-min approach is not equal to FAIR.

W1: I find it confusing that the paper at parts refers (e..g., at the start of Sec. 3.4) to its approach as a variable selection method. While I understand this phrasing for the previous works in which those algorithms only considers a subset of the available variables based on their relationship to the protected attribute $A$, the proposed algorithm considers all attributes, right? Under a variable selection problem, I would expect the algorithm to return a subset of attributes that satisfy counterfactual fairness the most, which is not the case in the experiments.

W2: It would help the paper to be more explicit about the limitations of the algorithm. For fairness, in particular, would it be able to handle more than one protected attribute? This is not discussed.