A Local Method for Satisfying Interventional Fairness with Partially Known Causal Graphs

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. We are also encouraged for the reviewer to recognize the Originality, Quality, Clarity, and Significance of our work. In below, we would like to address the reviewer's concerns regarding the Applicability of our proposed method from both theoretical and experimental aspects.

---

Main Result 1: The proposed method can also be applied in scenarios where sensitive attributes are identified as root nodes (W1).

• We agree with the reviewer that most sensitive attributes have no parents in nature. However, we respectively disagree that the proposed method relies on the existence of parent nodes of the sensitive attribute.

Theoretical Analysis

• For our studied interventional fairness, if the sensitive attribute $A$ is as a root node in the causal DAG, then there will be no backdoor path from the sensitive attribute $A$ to the outcome variable $Y$.

• At this point, the causal estimands in the definition of interventional fairness and our method degenerate to the correlation estimands.

• Compared to previous Fair, FairRelax, and $\epsilon$-IFair methods, our method doesn't rely on variable selection and strong causal assumptions, thus still should be considered as a more plausible method.

More Real-World Experiments

• We add experiments on two widely used real-world datasets: the Adult and the COMPAS datasets, where the sensitive attributes are "sex" and "race" as the root nodes, respectively.

• From the above experimental results, we find that our method still outperforms previous methods when the sensitive attribute is used as the root node.

More Synthetic Experiments with Varying Number of Parent Nodes

• To further explore the impact of the number of parent nodes of sensitive attributes on the accuracy and fairness performance, we also added more synthetic experiments with varying number of parent nodes.

• As the number of parent nodes increases, the advantage of our method over Fair, FairRelax, and $\epsilon$-IFair gradually increases. This is because when there are more parent nodes, there are more backdoor paths, thus our method can find these backdoor paths to better control the causal effect of the sensitive attribute on the outcome variable.

---

Main Result 2: The proposed method can also be applied to the Classification Problems (Q1).

Real-World Experiments

• For real-world experiments, we would like to kindly remind the reviewer that the outcome variable is binary in the OULAD dataset adopted in our original manuscript. The outcome variables in our added experiments on the Adult and COMPAS datasets are also binary.

Synthetic Experiments

• For synthetic experiments, as suggested by the reviewer, we conduct more experiments using AUC as the evaluation metric instead of RMSE. Specifically, we modify the data generation process (DGP) to clip the outcome variable $Y$ to 1 if $Y > 0$, and to 0 if $Y < 0$, and the rest DGP remains the same. The experiment results are shown below.

• From the above results, Full and Unaware perform better on AUC, while Fair, FairRelax, $\epsilon$-IFair, and our approach perform better on unfairness. Note that our approach outperforms Fair, FairRelax, and $\epsilon$-IFair in all scenarios on both AUC and unfairness.

---

(Minor) Clarity Issues (S3)

Clarity: The main results are summarized as lemmas. However, a lemma is usually used to help prove a larger theorem. The authors may change lemmas to propositions.

• We thank the reviewer for raising such useful suggestions. In our revised version, we have changed all Lemmas to Propositions.

---

We hope the above discussion will fully address your concerns about the applicability of our work, and we would really appreciate it if you could consider to raise your score. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

First, we clarify that methods for achieving fairness notions can be divided into the following three categories.

1. After careful thinking, we respectfully believe that it is not fair to compare our approach with methods for achieving correlation-based fairness in our degenerated case (in which the causal is the same as correlation), due to the following reasons.

• For methods achieving correlation-based fairness, their applicability usually fails to achieve causal fairness. However, our approach can achieve causal fairness with general applicability, where it degenerates to correlation-based fairness only in a specific case (in which the sensitive attribute is a root node in the causal DAG).
• The most important point, from the table above, is how do we determine if the sensitive attribute is the root node in the true causal DAG when we only have the observational data?
• We kindly invite the reviewer to refer to a toy example in Figure 4 on page 13 of our manuscript. Specifically, Figure 4(a) is the true causal DAG, in which the sensitive attribute is a root node. However, as shown in Figure 4(b), with only observation data, we cannot distinguish from the following causal relations: $X_1 \leftarrow A \to X_2$, or $X_1 \to A \to X_2$, or $X_1 \leftarrow A \leftarrow X_2$, because they are all in the same Markov equivalence class! But only the first causal relation $X_1 \leftarrow A \to X_2$ corresponds to the case where the sensitive attribute is a root node!

2. Meanwhile, we agree with the reviewer that it is fair to compare our approach with methods for achieving causal fairness without a known causal DAG, in terms of factors like effectiveness, applicability, etc., especially at the degenerated case.

To the best of our knowledge, Fair [1], FairRelax [1], and $\epsilon$-IFair [2] are all the existing prior works focusing on the same problem (causal fairness with partially known causal graph). To clarify the core idea, key assumptions, limitations, sensitivity to wrong CPDAG, and how they perform in the degenerated case in which the sensitive attribute as the root node, we summarize our findings as follows.

• In the degenerated case, we conclude that Fair [1] and FairRelax [1] perform poorly in both accuracy and fairness.
  • On one hand, they only use partial variables, not all, which reduces their accuracy performance.
  • On the other hand, after selecting the definite descendants (and possible descendants) of the sensitive attribute, they does not impose any causal fairness constraint, which reduces their fairness performance.
  • If the sensitive attribute is a root node in the causal DAG, most of the variables would be identified as the definite descendants (and possible descendants) via the causal discovery algorithms, which increases the availability of more variables and makes the unfair problem be more serious.
  • We also validate such claims on our added experiments on the Adult and the COMPAS datasets during rebuttal.
• In the degenerated case, both of the $\epsilon$-IFair [2] and our approaches can work well, but the time complexity for implementing $\epsilon$-IFair [2] in much larger than that of our approach.
  • This is because $\epsilon$-IFair [2] enumerates all subclasses of DAGs where the causal effect is identifiable, and when the sensitive attribute is the root node, the causal effect is naturally identifiable, so $\epsilon$-IFair [2] enumerates almost all possible DAGs.
  • For our local method, we only enumerate all possible parental sets of the sensitive attribute, whose enumerating space is much smaller than that of $\epsilon$-IFair [2], especially when we have plenty nodes in the real-world datasets.

---

We sincerely hope that the reviewer can carefully go over our discussion above -- we really spent a lot of time trying to make it clearer! We are also very grateful for the multiple rounds of communication with you, as this level of engagement has significantly contributed to improving the quality and positioning of our work (in the extensive fairness literature).

As the discussion deadline approaches, please let us know if you have any further feedback. We truly appreciate your beneficial comments as well as your consideration to upgrade your score -- thank you so much!!

---

References

[1] Counterfactual fairness with partially known causal graph. NeurIPS, 2022.

[2] Interventional fairness on partially known causal graphs: A constrained optimization approach. ICLR, 2024.

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. We are also encouraged by the very positive comments on our paper.

The paper's approach relies heavily on the accuracy of the CPDAG and on the generated propensity scores. The paper could address measures to address interventional fairness in the case of inaccuracies coming from the CPDAG.

As suggested by the reviewer, we added more simulation experiments to explore the effect of the degree of inaccuracy of CPDAG on the performance of our method. The experimental results are shown in Fig. 1 in the Supplementary PDF.

CPDAG was used on line 57 but was but the full meaning was not given until line 96.

We thank the reviewer for pointing out the typo. In our revised manuscript, we will provide the full name of CPDAG and more explanation at Line 57.

Flag For Ethics Review: Ethics review needed: Data privacy, copyright, and consent

We're guessing you misclicked the flag for ethics review. We would like to clarify that all used datasets in our original manuscript and rebuttal are public and widely adopted datasets.

• The OULAD dataset: https://www.archive.ics.uci.edu/dataset/349/open+university+learning+analytics+dataset

• The Adult dataset: https://archive.ics.uci.edu/dataset/2/adult

• The COMPAS dataset: https://www.kaggle.com/datasets/danofer/compass

---

We hope the above discussion will fully address your concerns about our work. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

The statement “One may notice that a function of X1 − X2 can be used to predict Y” appears somewhat abrupt and disrupts the flow of reading. A detailed explanation or a graphical illustration might clarify the independence between U1 − U2 and A.

Response: We thank the reviewer for the helpful comments. We will add the following toy example to illustrate such claims.

• Let $A = U_A, X_1 = A + U_1, X_2 = A + U_2$, and $Y = 2X_1 + X_2 + U_Y$.
• Because $Cov(X_1 – X_2, Y) \neq 0$, thus $X_1 – X_2$ is useful in predicting $Y$.
• Meanwhile, $X_1 – X_2 = U_1 – U_2$ is a constant for any value of $A$, so $U_1 – U_2$ is independent of $A$.
• Thus, using $X_1 – X_2$ as the variable for the prediction can still achieve counterfactual fairness.

At line 165, the rationale behind transforming certain undirected edges into directed edges in the opposite direction remains unclear. Elaboration on this would be beneficial.

Response: We would like to clarify the need for ‘transforming certain undirected edges into directed edges’ as follows.

• We know that causal discovery algorithms such as the PC algorithm can only return CPDAGs with undirected edges, not DAGs where all edges are directed.

• By definition, $sib(A)$ is a set containing all nodes that have an undirected edge with $A$, and our algorithm needs to find the set of all possible parent sets of the sensitive attribute $A$.

• Thus, we need to enumerate all possible cases (i.e., for some $Z \in sib(A), Z \rightarrow A$, and for other $Z \in sib(A), A \rightarrow Z$), and determine whether a new v-structure is generated.

• Formally, let $\mathbf{S}(A)$ be a subset of $sib(A)$, we can obtain a DAG from CPDAG by changing all undirected edges $\\{Z-A, \forall Z\in \mathbf{S}(A)\\}$ into the directed edges $\\{Z\to A, \forall Z\in \mathbf{S}(A)\\}$ as parents, and all of other undirected edges $\\{Z- A, \forall Z \not\in \mathbf{S}(A)\\}$ into the directed edges with opposite direction $\\{Z\leftarrow A, \forall Z \not\in \mathbf{S}(A)\\}$ as children.

The term ‘direct causal information set’ warrants a clear, explicit definition within the main text to avoid potential misunderstandings.

Response: The ‘direct causal information set’ is assumed to be the direct causal information in the form $X \to Y$, indicating that $X$ is a direct cause of $Y$. Such direct causal information are usually determined by the prior knowledge. Meanwhile, it is worth noting that other forms of background knowledge, such as tier orderings, specific model restrictions, or data obtained from previous experiments, can also induce MPDAGs [1-4]. As suggested by the reviewer, we will add the above explicit definition within the main text to avoid potential misunderstandings.

---

References

[1] Alain Hauser and Peter Buhlmann. Characterization and greedy learning of interventional Markov equivalence classes of directed acyclic graphs. JMLR, 2012.

[2] Marco F Eigenmann, Preetam Nandy, and Marloes H Maathuis. Structure learning of linear gaussian structural equation models with weak edges. ArXiv:1707.07560, 2017.

[3] Yuhao Wang, Liam Solus, Karren Yang, and Caroline Uhler. Permutation-based causal inference algorithms with interventions. NeurIPS, 2017.

[4] Dominik Rothenhausler, Jan Ernest, and Peter Buhlmann. Causal inference in partially linear structural equation models. The Annals of Statistics, 2018.

---

We hope the above discussion will fully address your concerns about our work. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. In below, we address these concerns point by point and try our best to update the manuscript accordingly.

Weaknesses

Above line 188 : It is unclear what $S^{(i)}(A) ; 1<=i<=M$ refers to. The authors should explain the superscripts.

• We know that causal discovery algorithms can only return CPDAGs with undirected edges, not DAGs where all edges are directed.

• Let $sib(A)$ be a set containing all nodes that have an undirected edge with $A$, and our algorithm needs to find the set of all possible parent sets of the sensitive attribute $A$.

• $S^{(i)}(A) ; 1<=i<=M$ is the subsets of $sib(A)$, such that $\mathcal{S}_A=\\{p a(A)\cup \mathbf{S}^{(1)}(A), \ldots, p a(A)\cup \mathbf{S}^{(M)}(A)\\}$ consists of all possible parent sets of the sensitive attribute $A$ obtained from our algorithm.

Equations should be numbered. The Equation after line 192 should be explained in more detail.

• The propensity $P(A \mid p a(A)\cup \mathbf{S}^{(m)}(A))$ is defined as the probability of the (binary) sensitive attribute $A$ given its (possible) parent set $pa(A)\cup \mathbf{S}^{(m)}(A)$.

• The Equation after line 192 is the binary classification cross-entropy loss for training the propensity model $g(X; \hat \phi^{(m)})$ for estimating the true propensity $P(A \mid p a(A)\cup \mathbf{S}^{(m)}(A))$.

• The input is covariates $X$ restricted on the possible parent set $p a(A, \mathcal{H})\cup \mathbf{S}^{(m)}(A)$, denoted as $X|_{p a(A, \mathcal{H})\cup \mathbf{S}^{(m)}(A)}$.

Line 201: The authors should provide some intuition/explanation about the backdoor adjustment formula.

• Whenever we undertake to evaluate the effect of one factor $A$ on another $Y$, the question arises as to whether we should adjust our measurements for possible variations in some other factors $X$, otherwise known as "confounders".

• The backdoor adjustment formula aims to partitioning the population into groups that are homogeneous relative to $X$, assessing the effect of $A$ on $Y$ in each homogeneous group, and then averaging the results.

• Formally, the backdoor adjustment formula is $P(Y=y \mid do(A=a))=\sum_x P(Y=y \mid A=a, X=x) P(X=x)$, which computes the association between $A$ and $Y$ for each value $x$ of $X$, then averages over those values, also referred as "adjusting for $X$".

It seems that there are no baselines based on prior works. Is there no work that solves the fairness problem with causal effect estimation?

• For compared baselines, we would like to clarify that FairRelax [R1], Fair [R1], and $\epsilon$-IFair [R2] are all prior works solving the same problem (causal fairness with partially known causal graph). We will explicitly add citations in our revised "Baselines" part.

• To the best of our knowledge, we have compared the most comprehensive and SOTA methods, e.g., [R2] is published in ICLR 24.

• The important reason for the limited baselines is because most previous causal fairness work requires known causal graphs (DAGs), whereas we only need tabular data (from which we can only obtain CPDAGs).

Questions

Line 178: why can it not be $X_i \leftarrow A \to X_j$?

• From Definition 3.1 in Line 157, the v-structure is defined as $X_i \to A \leftarrow X_j$, not $X_i \leftarrow A \to X_j$.

• From Lemma 3.2 in Line 170, a set $\mathbf{S}(A) \subset sib(A)$ is a possible parent set of the sensitive attribute $A$, if and only if there is no more v-structures.

• Thus, as stated in Line 178, if $X_i$ and $X_j$ are not adjacent, they cannot be both in the parent set of $A$. Otherwise, a new v-structure $X_i \to A \leftarrow X_j$ is formed.

What is the computational complexity of the proposed approach?

• First, our proposed approach mainly includes the following 3 steps: constructing the CPDAG from the tabular data, estimating all possible propensities (Alg. 1), and learning fair classifier via minimax approach (Alg. 2).

• From [R1], the complexity of constructing the CPDAG in the worst case is $\mathcal{O}(|sib(S, \mathcal{G})+ch(S, \mathcal{G})| *|E(\mathcal{G})|)$, where $|E(\mathcal{G})|$ is the number of edges in $\mathcal{G}$.

• For Alg. 1 and Alg. 2, the proposed local method needs to identify the v-structures for every node pairs in $sib(S, \mathcal{G})$, thus the computational complexity is $\mathcal{O}(|sib(S, \mathcal{G})|^2)$.

• Thus, the overall computational complexity is $\mathcal{O}(|sib(S, \mathcal{G})+ch(S, \mathcal{G})| *|E(\mathcal{G})|+|sib(S, \mathcal{G})|^2)$, which also illustrates the advantage of our local method instead of enumerating on all possible DAGs.

How can the proposed approach be connected with the min-max approach proposed by Xia et al [1] and the invariant prediction problem proposed by Subbaswamy et al [2]?

• For the min-max approach proposed by Xia et al [1], one similar point is that the input is also the observational data, rather than requiring the known DAG.

• They found that a neural net is unable to predict the effects of interventions given observational data alone (Thm. 1), which motivates to introduce a special type of SCM called a neural causal model (NCM) for performing causal inferences.

• However, our paper still aims to conduct the causal inferences in the context of general SCM, with minimax approach for bounding the uncertainties of the true DAGs.

• For the invariant prediction problem proposed by Subbaswamy et al [2], they aims to solve the dataset shift problem with different training and target distributions.

• However, our study aims to achieve causal fairness without the explicit knowledge of causal DAGs, with same training and target distributions.

---

References

[R1] Counterfactual fairness with partially known causal graph. NeurIPS, 2022.

[R2] Interventional fairness on partially known causal graphs: A constrained optimization approach. ICLR, 2024.