Procedural Fairness Through Decoupling Objectionable Data Generating Components

Thank you for the thoughtful questions and comments, and for the time devoted! Below please see our responses to specific points in the review comments:

---

C1: "There are only two experiments, one of which is a synthetic dataset."

R1: Thanks for the comment. In addition to the two sets of experiments on simulated data and the UCI Adult data set, we have included additional experimental results on the real-world Folktables data set (Ding et al., 2021). In particular, we consider the public health coverage prediction task where there are 17 features and 1 target variable. In light of the comment, we have included the additional experimental results in Appendix D.4, along with the side note Re: C1 by Reviewer R8GC on page 23.

---

Q2: "Does your framework require a causal model between attributes?"

R2: Thanks for the question on the causal model. Yes, our framework utilizes the causal model between attributes. Our assumption on the availability of the causal model (e.g., in terms of a causal graph) is consistent with previous literature on causal fairness. For the convenience of locating the related material, we provide the side note Re: Q2 by Reviewer R8GC on page 5.

---

Q3: "Are their simpler toy examples which you can illustrate as a warm-up?"

R3: If we understood it correctly, the question is about the illustrative example of our framework. Yes, we presented a worked-out example in Appendix C to illustrate in detail Algorithm 1 and Algorithm 2 of our framework. For the convenience of locating the related material, we provide the side note Re: Q3 by Reviewer R8GC on page 21. Please kindly let us know if we misunderstood the question.

---

Q4: "In practice, what does it mean to constrain the data generating process (DGP)? Are we essentially filtering out data based on constraints before learning a model?"

R4: Thanks for the question. By "constraint" on DGP we are referring to the fairness constraints on certain causal quantities proposed in previous causal fairness literature. We are not filtering out data based on such constraints before learning a model. Instead, previous causal fairness notions propose that the causal effect of interest should be bounded. For completeness, we presented the literature review on causal fairness notions in Appendix A.1. We provide the side note Re: Q4 by Reviewer R8GC on page 17.

---

Q5: "Are there other examples of work which look at the DGP and constraining it? Why haven't they?"

R5: Thanks for the question about previous causal fairness notions. Yes, there are previous works that look at DGP and constrain it. For instance, Counterfactual Fairness (Kusner et al., 2017) requires that the counterfactual causal effect between the protected feature and the prediction should be bounded by a small positive constant. We presented the literature review on causal fairness notions in Appendix A.1, and provided discussions on differences and connections of our approach compared to previous causal fairness works in Appendix B.1. We provide the side note Re: Q5 by Reviewer R8GC on page 19.

---

Reference

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In Advances in Neural Information Processing Systems, volume 34, pp. 6478–6490, 2021.

Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Neural Information Processing Systems, pp. 4066–4076, 2017.

Thanks for the thoughtful and detailed comments, as well as the time and effort devoted! Below please see our responses to specific comments and questions:

---

Q1: "Is there a practical way to identify objectionable and neutral components in high-dimensional settings with intricate and unknown dependencies between variables?"

R1: Thanks for the thoughtful question about the availability of the causal model and the specification of objectionable and neutral components! With the recent developments in the causal discovery literature, under certain mild assumptions, the causal graph can be recovered from high-dimensional settings. Since causal discovery (e.g., to find the causal graph) and algorithmic fairness (to define/quantify discrimination given a causal model) are different tasks, we follow previous causal fairness literature and assume the availability of the causal graph. It has been recognized in the literature that the fairness pursuit is not a pure technical question, and that it requires expert knowledge from different domains. Therefore, our framework takes the specified objectionable and neutral components, together with the causal graph, as part of the input. We reveal and address the often-overlooked disguised procedural unfairness, and show that even if the causal graph is available, previous causal fairness notions can still violate the requirements of procedural fairness.

---

C2: "While the optimization problem of configuring reference point values to maximize the benefits for the least advantaged individuals makes sense, it is unclear how complex it is."

R2: Thanks for carefully considering the detail of our approach! In addition to our code implementation in the supplementary material (we provided the anonymous link in the paper), we have also included descriptions of implementation details in Appendix D.1.1, and the illustration (with more than 1000 variables) and discussion on scalability of our approach in Appendix D.1.2. For the convenience of locating the related material, we provide the side note Re: C2 by Reviewer GSnz on page 24.

---

C3: "The evaluation is based on the UCI Adult dataset only, and the results might not be generalizable across different domains or datasets."

R3: Thanks for the question about experimental results on other data sets. In light of your comment, we have included additional experimental results on the real-world Folktables data set (Ding et al., 2021). We consider the public health coverage prediction task, and present the additional experimental results in Appendix D.4, along with the side note Re: C3 by Reviewer GSnz on page 28.

---

C4: "Also, a comparison to other fairness strategies might be insightful."

R4: Thanks for sharing the suggestion. We completely agree, and this is exactly why we presented literature review over different fairness strategies, including causal fairness notions (Appendix A.1), scenarios with responsive agents (Appendix A.2), and fair representation learning (Appendix A.3). We also discussed in detail differences and connections of our approach compared to previous fairness strategies (Appendix B) and summarized the comparison in Table 1. For the convenience of locating the related material, we provide the side note Re: C4 by Reviewer GSnz on page 17.

---

Reference

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In Advances in Neural Information Processing Systems, volume 34, pp. 6478–6490, 2021.

We are very grateful for your thoughtful and detailed comments, as well as the time and effort devoted!

Following your constructive suggestions, we have incorporated following changes in our revision:

1. We include descriptions of our implementing details, which serve as complements to our code implementation provided in the supplementary material.

2. We provide a quantitative demonstration of the scalability of our framework. We consider a scenario involving more than 1,000 variables to illustrate the computational costs.

3. We include additional experimental results on the real-world data on the public healthcare prediction task.

4. We provide the pointer to the comparison to other fairness strategies, including the literature review and the detailed discussions on differences and connections.

We have also provided color-coded side notes in the revised manuscript to help locate the related material and discussions.

We are very happy to get to know that you found our approach "well-motivated and well-presented", and that the problem we address "is interesting". We sincerely hope our further clarifications can fully address the questions on our framework implementation, and can be helpful in the evaluation of our results. We are eagerly looking forward to your kind feedback.

We are very grateful for your constructive and insightful comments, and for the time and effort devoted! Below please see our responses to specific points in the review comment:

---

C1: "It's unclear how broadly the framework can be applied across different domains and whether there are any limitations to its scalability."

R1: Thanks for considering the applicability and scalability of our approach! Our framework can be applied to linear and nonlinear data generating processes, and scales well with the number of variables. To demonstrate the scalability of our framework, we have included a quantitative illustration of the computational cost (with more than 1,000 variables) in Appendix D.1.2. The computational cost of our approach is comparable to that of the vanilla regression model (implemented with a two-hidden-layer neural network). For the convenience of locating the related material, we provide the side note Re: C1 by Reviewer 2K3E on page 24.

---

C2: "[The reviewer] did not find a discussion regarding the practicality of implementing the proposed framework."

R2: Thanks for carefully considering the practicality of implementing our framework! In addition to our code implementation in the supplementary material (we provided the anonymous link in the paper), we have also included descriptions of our implementation details in Appendix D.1.1, along with the side note Re: C2 by Reviewer 2K3E on page 24.

---

Q3: "Have you tried to use the proposed framework to more complex datasets (more variables and larger domains)?"

R3: Thanks for the question on experiments with additional data. In addition to the two sets of experiments on simulated data and UCI Adult data set, we have included additional experimental results on the real-world Folktables data set (Ding et al., 2021). In particular, we consider the public health coverage prediction task where there are 17 features and 1 target variable. In light of the comment, we have included the additional experimental results in Appendix D.4, along with the side note Re: Q3 by Reviewer 2K3E on page 28.

---

Q4: "What happens when you have multiple confounding variables?"

R4: Thanks for the thoughtful question on confounding variables! We believe the term "confounding variables" in the question refers to the (potentially latent) confounders that connect with observed nodes via direct edges coming into the nodes. Our value instantiation rule (Algorithm 1) still works when there are confounders. Among the three options (Steps 5 -- 10 of Algorithm 1), the reference point value configuration focuses on the tail node of the objectionable components (edges going out of the tail node). The other two options, namely, the downstream of reference points and the original value in data, naturally adapt to the scenario with confounders, since we propagate the information from causal upstream to causal downstream. We provide the side note Re: Q4 by Reviewer 2K3E on page 6. Please kindly let us know if we misunderstood the question.

---

Q5: "What are the computational costs associated with implementing the framework, and how do they compare to existing methods?"

R5: Thanks for the question about computational costs. The decoupling of objectionable data generating components does not require retraining the modeling for local causal modules. The computational overhead only comes from the inference phase of the model during the simulated annealing process (to optimize reference point configurations). The computation cost of our approach is comparable to that of existing methods based on constrained optimization. In light of the question, we have included the quantitative illustration as well as the above discussion on computational costs in Appendix D.1.2, along with the side note Re: Q5 by Reviewer 2K3E on page 24.

---

Reference

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In Advances in Neural Information Processing Systems, volume 34, pp. 6478–6490, 2021