Response to “Questions"

1. Definition of action fairness: Thank you for critically questioning our definition of action fairness and for allowing us to elaborate on our definition. You are correct that in our definition of action fairness ($\pi(X) \perp S$), our policy $\pi$ is a function of all confounders $X$, i.e., also the ones that may be spuriously correlated with $S$. However, enforcing the independence statement $\pi^{\mathrm{af}}(X) \perp S$ ensures that the policy $pi$ does not depend on such spuriously correlated confounders. To see this, let us consider the example you suggested, where $X_0 = f(S)$ for some (measurable) function $S$ and $X_{1:m}$ are independent of $S$. This implies that $\pi(X) = \pi(f(S), X_1, \dots, X_m)$ is a (measurable) function of $S$. Hence, the independence $\pi(f(S), X_1, \dots, X_m) \perp S$ only holds if either $f$ is constant (in which case $X_0$ and $S$ are independent), or if $\pi(\cdot, x_1, \dots, x_m)$ is constant for all x_1, \dots, x_m (almost surely). We agree that the definition you provided would be another valid way of defining action fairness. However, the benefit of our definition is that we do not assume prior knowledge on $X$ (e.g., which components are independent of $S$).

   Action: We added a clarification to Sec 4.1 along the above lines.

2. Pareto optimal policies: In our paper, we go beyond existing work from Viviano and Brandic (VB) in several ways: (i) We consider different fairness notions than VB (i.e., action fairness, value fairness, and interactions thereof); (ii) VB maximize fairness over the set of Pareto optimal policies. Here, Pareto optimality is defined so that the policy value of one sensitive group cannot be improved without reducing the policy value for the opposite group. In contrast, we propose to incorporate our fairness notions by adjusting the off-policy learning objective (value fairness), and then maximize this objective over the class of action fair policies. (iii) While VB is restricted to learning linear policies, our approach (outlined in (ii)) enables learning arbitrarily non-linear policies (which we refer to as “general policies”). This is possible because we can incorporate the action fairness constraint by leveraging fair representation learning to obtain a representation independent of the sensitive attribute, which can be used in a second step to train an optimal policy. Notwithstanding, the VB approach can not work with non-linear policies and uses different fairness notions, hence, is not applicable in our case.

   Action: We clarified the definition of “Pareto-optimality” and “general policy” in our related work and elaborated on the differences in our paper.

3. Experimental details: In Table 1, we reported the results for all methods $\mathrm{m} \in \{\mathrm{DM}, \mathrm{IPW}, \mathrm{DR}\}$. Here, for each method, we reported three rows corresponding to $m$. In particular, we do not observe large differences in results between the three methods. Thus, we fixed $m = \mathrm{DR}$ (corresponding to the doubly robust method) in our other experiments (e.g., Table 2; Fig. 3. and 4.).

   Action: We added a clarification to Sec. 6.

Response to “Other suggestions”

• “Outcome regression”: Thank you for pointing out this ambiguity. Action: We replaced the naming “outcome regression” with “response surface”, which is consistent with previous work on off-policy learning and treatment effect estimation.
• User vs action covariates: Thank you. You are correct in that we defined our covariates $X$ to be user-specific attributes, not attributes of the action/treatment. Action: We added a clarification to Sec. 3.

Thank you for your comprehensive and actionable review! We took all your comments at heart and improved our paper as follows:

Response to “Weaknesses”

1. Formulations and mathematical descriptions: Thank you for pointing this out. Please refer to our “Response to Questions”.

2. Related work on fair representation learning: We kindly refer to Appendix A.1, in which we provide a brief overview of related fair representation learning. We would like to emphasize that our main contribution is not a new method for fair representation learning. Rather, we adapt fairness notions and provide an understanding of how these fairness notions interact in the context of off-policy learning, while we at the same time propose a practical framework to learn fair optimal policies and provide theoretical guarantees in the form of generalization bounds. However, we acknowledge that the literature stream on fair representation learning deserves a more thorough treatment in our related work.

   Action: We expanded our related work on fair representation learning and moved it to the main paper (see Sec. 2, new paragraph “fair representation learning”). Furthermore, we added new experiments using alternative approaches for fair representation learning (see our new Appendix K). However, we emphasize that fair representation learning is orthogonal to our contribution as we develop a new framework on top of it for fair off-policy learning.

3. How is the problem of off-policy learning resolved? Thank you for giving us the opportunity to clarify this point. You are correct that one of the key challenges in combining off-policy learning with our fairness constraints is the problem of estimating the conditional policy values $V^\mathrm{m}_s(\pi)$. In our paper, we proposed unbiased estimators $\hat{V}^\mathrm{m}_s(\pi)$ (see Sec. 5.2.), which can be used in combination with any standard off-policy learning method $\mathrm{m} \in \{\mathrm{DM}, \mathrm{IPW}, \mathrm{DR}\}$. In our experiments, we provided ablation studies in the sense that we applied our framework with all three methods $m$ (see e.g., Table 1). Furthermore, we provided theoretical guarantees for the finite-sample errors of our fairness objective estimators in the form of generalization bounds (Theorem 1), which depend on the method $m$ (through the constant $K_m$). Hence, we provided both theoretical and empirical insights for solving the off-policy learning problem in our fairness context.

   Action: We clarify the above in both Sec. 5.2. and Sec. 5.3. Furthermore, we emphasize the dependence of the bounds in Theorem 1 on the off-policy learning method $m$.

4. Results on real-world data: In Table 2, we first report the results for our framework with only action fairness and without value fairness (second row, “FairPol only with action fairness”). Here, value fairness is indeed worse compared to the unrestricted policy (first row). However, this is to be expected as FairPol enforces action fairness, which can worsen value fairness (we refer to our toy example in Appendix C. for a detailed discussion on this issue). In contrast, the (estimated) action fairness is improved by a large margin. Furthermore, comparing our results for value fairness (third and fourth row of Table 2) directly with the unrestricted policy (first row) is misleading because the unrestricted policy does not ensure action fairness. Instead, FairPol with max-min fairness (fourth row of Table 2) is effective in improving value fairness compared to FairPol with only action fairness (second row of Table 2). In summary, the results using real-world data confirm the effectiveness of our framework.

   Action: We followed your suggestion and expanded our discussion of our results using real-world data in Sec. 6.

5. Clarity of writing: Thank you for pointing this out. We improved our writing along your suggestions. Please refer to our “Response to Questions”.

Thank you for the detailed response and for sharing the revision. The revision on the discussion of fair representation learning and comparison with (Viviano and Brandic) would be helpful for readers.

I have several remaining comments on rebuttals and revisions.

---

• According to the call for papers, we have a strict page limit of 9 pages. However, the current draft is over this page limit. How do you plan to fit the paper to this limit?

There will be a strict upper limit of 9 pages for the main text of the submission, with unlimited additional pages for citations. This page limit applies to both the initial and final camera ready version. https://iclr.cc/Conferences/2023/CallForPapers

---

• About Weakness and Question 1

1. Definition of action fairness: Thank you for critically questioning our definition of action fairness and for allowing us to elaborate on our definition. You are correct that in our definition of action fairness ($\pi(X | S)$), our policy $\pi$ is a function of all confounders $X$, i.e., also the ones that may be spuriously correlated with $S$.

In this case, the following sentence in the current draft should be mathematically incorrect, as I pointed out in the initial review. This part should be revised to avoid readers' confusion about math in this paper.

a policy $\pi$ should assign the same action $A = A'$ to two individuals with the same covariates $X=X'$ but different gender $S \neq S'$.

I am not pointing out the description of Definition 1. Rather, the concept of describing action fairness (the above sentence) is contradictory to the description in Introduction.

We agree that the definition you provided would be another valid way of defining action fairness. However, the benefit of our definition is that we do not assume prior knowledge on $X_0$ (e.g., which components are independent of $S$).

In addition, I do not suggest assuming prior knowledge on $X_0$ here. My point is that the proper description, that aligns with Introduction or some other sentences in the paper, should need $X_0$, regardless of assuming prior knowledge on $X_0$.

---

• About suggestion and revision

“Outcome regression”: Thank you for pointing out this ambiguity. Action: We replaced the naming “outcome regression” with “response surface”, which is consistent with previous work on off-policy learning and treatment effect estimation.

I was not commenting on the terminology but was pointing out the improper use of math. By definition, $\mathbb{E}[Y | X, S, A=j]$ is by the expected outcome conditioned on $X, S, A=j$. Therefore, if the paper denotes $\mu_j(X, S) = \mathbb{E}[Y | X, S, A=j]$, $\mu_j(X, S)$ cannot be the regressed outcome (because it is the "true" expected outcome by definition) in DM and DR. Though I haven't checked all the math descriptions, the math in this paper seems doubtful to me, given that there are already two misrepresentations.

---

• About theoretical analysis.

This is a clarification question, where are $\nu$ and $\eta$ defined in Theorem 1? (It seems they are not defined.)

Furthermore, we provided theoretical guarantees for the finite-sample errors of our fairness objective estimators in the form of generalization bounds (Theorem 1), which depend on the method $m$ (through the constant $K_m$).

In addition, several things remain unclear, e.g., (i) how is the source of difference in the generalization bound among DM, IPS, and DR? In other words, in what situation does the difference emerge? (ii) How the generalization bound can be different when considering fairness compared to standard analysis of the generalization bound of DM, IPS, and DR? (iii) How to interpret three results of generalization bounds? It would be great if the presentation of the theoretical analysis could be improved so readers can understand the connections and differences with the standard OPL papers.

---

• About the experiment.

we provided both theoretical and empirical insights for solving the off-policy learning problem in our fairness context.

Typical OPL papers (e.g., Swaminathan & Joachims, 15) often compare the methods with varying data sizes to confirm the theoretical results of generalization bound analysis, while not provided in this paper.

Overall (not limited to the experiment section), the discussion of challenges in using observational data still seems thin from the manuscript despite being one important topic of this paper.

Swaminathan & Joachims, 15: Adith Swaminathan and Thorsten Joachims. "The Self-Normalized Estimator for Counterfactual Learning." NeurIPS, 2015.

Thank you for your quick response and for allowing a constructive discussion. In the following, we address your remaining concerns and are confident that this improved our paper further.

• Page limit: Thank you for pointing this out. We were unsure whether the page limit applies during the rebuttal, and, to address your comment, we uploaded a revised version with exactly 9 pages.

• Action fairness: We apologize for the misunderstanding. You are correct, our sentence in the introduction does not fit our formalization of action fairness. Action: We changed our introduction to action fairness at the beginning of Sec. 4. to fit our formal definition.

• Outcome regressions/response surfaces: You are correct that response surfaces $\mu_j(X, S)$ denote the ground-truth expectation of $Y$ conditioned on $X, S, A = j$. This is deliberate because the response surfaces $\mu_j(X, S)$ do not denote estimated quantities. However, this conditional expectation can be estimated from the observational data (e.g., by performing a regression of Y on X, S, A.). Note that the same holds for the ground-truth behavioral policy $\pi_b$, which reduces to a classification problem for binary treatments. In our experiments, we use a TARNet [1] to estimate the nuisance parameters (i.e., $\mu_j$ and $\pi_b$). However, our framework is agnostic to the nuisance estimator and could be used in combination with any other estimator (e.g., S/T learner [2]).

  Hence, we emphasize that all our mathematical statements are correct. Please note that off-policy learning/ treatment effect estimation under nuisance parameters is a well-studied topic and that we are not the first who propose this type of notation [2, 3,4]. Is your concern potentially that, in Theorem 1, we do not account for estimation errors in the nuisance parameters (because we use the ground-truth nuisance parameters)? While this is true, it is not a major concern because it has been shown that (under uniform consistency of nuisance estimators), estimation errors in the nuisance parameters do not affect the leading term in the convergence rate (see Sec. 3.1, Sec.3.2, Lemma 4 in [4]). This is also consistent with previous work that provides generalization bounds for off-policy learning [5].

.

Response to reviewer TRGr

Thank you for your review and your helpful comments!

Response to “Weaknesses”

• Thank you for giving us the opportunity to clarify our technical contributions. You are correct in that we build our framework on existing literature streams, e.g., fair representation learning. However, we believe that our paper makes several non-trivial technical contributions beyond that. In particular: (i) We adapt fairness notions from related literature and formalize these for off-policy learning from observational data. (ii) We provide a theoretical understanding of how these fairness notions interact in the context of off-policy learning (see Lemma 1 and 2, as well as our toy examples in Appendix C and D). (iii) We propose a practical framework to learn fair optimal policies. While we build upon ideas from fair representation learning to achieve action fairness, we combine these with the off-policy learning methodology and provide novel theoretical guarantees in the form of generalization bounds (Theorem 1). Hence, we believe that our paper contains several technical contributions that add to the state-of-the-art in algorithmic fairness and off-policy learning in a non-trivial manner.

  Action: We clarified our technical contributions in Sec.1. We also expanded our discussion (see end of Sec. 6), in which we also reiterated our contributions

• No baselines available: We agree that, generally, a thorough comparison with baselines is beneficial to provide a rigorous empirical evaluation. Unfortunately, there are no baselines directly available as our setting of fair policy learning has not yet been studied before.
  Nevertheless, we took great strides to adapt existing methods as baselines. In the context of our work, we see two potential literature streams that could provide methods baselines for our experiments: (i) Methods for standard off-policy learning from observational data (not considering fairness); and (ii) methods that incorporate fairness notions into off-policy learning. In the following, we lay out how we build upon methods from (i) and (ii) for benchmarking:

  • For (i), the following three methods are considered standard in the literature [1]: the direct method, the IPW method, and the DR method. **Our proposed framework works for all of these, and we provided experiments for all three baselines **(see Table 1). We did not observe large differences in performance.
  • For (ii), we are not aware of any other work that aims at achieving the same fairness notions (i.e., combining action and value fairness) in the context of off-policy learning. Any potential benchmarking experiment with existing methods would thus involve comparing methods that satisfy different fairness notions. Hence, we follow established fairness literature [e.g., 2, 3] and do not compare methods targeted at different fairness notions. Instead, we investigated in our experiments how well our proposed method achieves our proposed fairness notions, and how this affects the performance (policy value). For this purpose, we provided several ablations studies: for example, we remove parts of our method (value fairness), and we compare with an unrestricted policy that does not satisfy any fairness notions.

  We agree that our approach for fair representation learning (adversarial domain confusion loss) could in principle be replaced by any other approach that aims at enforcing independence from the sensitive attribute. Hence, we provide an additional ablation study (see new Appendix K), where we compare the first step of our framework with two other approaches: (i) Adversarial learning using gradient reversal; and (ii) regularization using a Wasserstein distance. Our chosen approach (adversarial domain confusion loss) performs best, which is also consistent with prior literature [1].

  Action: We added new experiments based on fair representation learning (see our new Appendix K).

Thank you for your positive evaluation of our paper! We took all your comments at heart and improved our paper accordingly.

Response to “Weaknesses”

• No baselines available: We agree that, generally, a thorough comparison with baselines is beneficial to provide a rigorous empirical evaluation. Unfortunately, there are no baselines directly available as our setting of fair policy learning has not yet been studied before.
  Nevertheless, we took great strides to adapt existing methods as baselines. In the context of our work, we see two potential literature streams that could provide methods baselines for our experiments: (i) Methods for standard off-policy learning from observational data (not considering fairness); and (ii) methods that incorporate fairness notions into off-policy learning. In the following, we lay out how we build upon methods from (i) and (ii) for benchmarking:

  • For (i), the following three methods are considered standard in the literature [1]: the direct method, the IPW method, and the DR method. Our proposed framework works for all of these, and we provided experiments for all three baselines (see Table 1). We did not observe large differences in performance.
  • For (ii), we are not aware of any other work that aims at achieving the same fairness notions (i.e., combining action and value fairness) in the context of off-policy learning. Any potential benchmarking experiment with existing methods would thus involve comparing methods that satisfy different fairness notions (and thus have entirely different objectives that are incompatible and thus not comparable).
    Hence, we follow established fairness literature [e.g., 2, 3] and do not compare methods targeted at different fairness notions. Instead, we investigated in our experiments how well our proposed method achieves our proposed fairness notions, and how this affects the performance (policy value). Hence, we provided several ablations studies: for example, we removed parts of our method (value fairness), and we compared with an unrestricted policy that does not satisfy any fairness notions.

  Nevertheless, we understand that you ask for a more extensive numerical evaluation.

  Action: Our approach for fair representation learning (adversarial domain confusion loss) can in principle be replaced by any other approach that aims at enforcing independence from the sensitive attribute. Hence, we provide a new ablation study (see new Appendix K), where we compare the first step of our framework with two other approaches: (i) Adversarial learning using gradient reversal; and (ii) regularization using a Wasserstein distance. Our chosen approach (adversarial domain confusion loss) performs best, which is also consistent with prior literature [4].

• Limited discussion: Thank you. We followed your suggestion and, to improve our paper, now offer an extended discussion.

  Action: We followed your suggestion and moved parts of Appendix I to our new Sec. 7. In particular, we now discuss our contribution and provide an outlook on future research directions.

• Readability of plots: Thank you; we have improved our plots. Action: We followed your suggestion and increased both line thickness and font size in Fig.3 and 4.

References

[1] Athey and Wager (2021). “Policy learning from observational data” Econometrica.

[2] Madras et al. (2018). “Learning adversarially fair and transferable representations” ICML

[3] Craeger et al. (2019) “Flexibly Fair Representation Learning by Disentanglement” ICML

[4] Melnychuk et al. (2022) "Causal transformer for estimating counterfactual outcomes" ICML

Thank you for your review and your helpful comments. As you can see below, we have carefully revised our paper along with your suggestions.

Response to “Weaknesses”

1. Observational data/ nuisance parameters: Thank you for your question. You are correct that our method (or rather any method for off-policy learning) depends on the policy score $\psi^\mathrm{m}(\pi, \mathcal{D})$ and, therefore, on the unknown nuisance parameters $\mu_1$, $\mu_0$, and $\mu_0$ (which correspond to the choice of $\mathrm{m} \in \{\mathrm{DM}, \mathrm{IPW}, \mathrm{DR}\}$, respectively). Under Assumption 1, these nuisance parameters can be estimated from the observational data using arbitrary regression/machine learning algorithms (see Sec. 3). In our experiments on synthetic data, we do not focus on the question of estimation errors in the nuisance parameters but rather aim to explore the impact of our proposed fairness notions. For this purpose, we refrain from estimating the nuisance parameters but instead use the ground-truth nuisance parameters, which are known in our simulations (we refer to Appendix G for details). Note that we do this for all methods to ensure a fair comparison of the results.

   Motivated by your comment, we added new experimental results (see our new Appendix L). Therein, we re-ran our experiment from Table 1 but now use TARNet as a state-of-the-art method [1] to estimate nuisance parameters (instead of using the ground-truth). This allows us to analyze the robustness of our results on synthetic data with respect to nuisance errors. Our new results show that our results remain robust when using estimated nuisance parameters.

   Action: We provided a more elaborate explanation for our evaluation. Furthermore, we added new experimental results by using TARNet for estimating nuisance parameters instead of using the ground truth (see our new Appendix L). We added details regarding the implementation of TARNet to Appendix M.

2. Empirical evaluation on real-world data: Thank you for your question. You are correct that we do not observe the counterfactual outcomes on the real-world data, which means that we can not evaluate our methods using oracle metrics (as in Table 1). As mentioned in the previous point, we use a TARNet to estimate the nuisance parameters on real-world data. Thus, we can plug the estimated nuisance parameters into our (conditional) policy value estimators $\hat{V}^\mathrm{m}(\pi)$ and $\hat{V}^\mathrm{m}_s(\pi)$ (proposed in Sec. 5.2.). This is what we report in, e.g., Table 2. Of note, our evaluation is in line with works on treatment effect estimation, which often report the factual MSE as a heuristic to measure performance on real-world data [1]. In principle, our policy value estimators may be biased if not all confounders are collected in the data. However, in our experiments, we do not aim to compare performance (e.g., in terms of unbiased estimation), but rather aim to investigate the impact of our fairness notions on our learned policy.

   For action fairness, we reported the Spearman correlation between policy predictions and the sensitive attribute, which enables detecting (potentially nonlinear) dependence.

   Action: We added a clarification to Sec. 6.

3. Contribution/novelty of fairness notions: Thanks. You are correct in that our proposed fairness notions are adapted from existing literature streams. For example, “action fairness” resembles “demographic parity” in standard fair machine learning, and both “max-min” and “envy-free” fairness have also been proposed in works from optimization (e.g., from resource allocation). The reason is that the fairness notions are deeply grounded in distributive justice. Please note that we do not claim to be the first to propose the mentioned fairness notions. Instead, our contributions are as follows:

   1. (i) We propose to adapt existing fairness notions from related literature (e.g., resource allocation), but formalize these for off-policy learning from observational data;
   2. (ii) We provide a theoretical understanding of how these fairness notions interact in the context of off-policy learning (see Lemma 1 and 2, as well as our toy examples in Appendix C and D);
   3. (iii) We propose a practical framework to learn fair optimal policies and provide theoretical guarantees in the form of generalization bounds.

   Action: We clarified our contributions in Sec.1. We also expanded our discussion (see end of Sec. 6), in which we also reiterated our contributions