Learning Counterfactual Outcomes Under Rank Preservation

Thank you very much for your positive evaluation of our paper. Below, we hope that our clarification addresses your concerns.

W1: The paper's core contribution relies on the rank preservation assumption. While the introduction briefly explains it, it lacks a discussion or illustrative examples to motivate why this assumption is appropriate and practical in real-world scenarios. Adding this would significantly strengthen the paper's introduction.

Response to W1: Thank you for your helpful suggestions. We first provide more explanations for Assumption 4.2 ($\rho(Y_x, Y_{x'}\mid Z)=1$). Assumption 4.2 implies that an individual’s factual and counterfactual outcomes have the same rank in the corresponding distributions of factual and counterfactual outcomes for all individuals. To better understand this, imagine two counterfactual worlds:

• In the first world, every individual receives treatment $x$;
• In the second world, every individual receives treatment $x'$.

For a given individual $i$, their outcomes in these two worlds are $y_{i, x}$ and $y_{i, x'}$, respectively. Assumption 4.2 states that the rank of $y_{i, x}$ among all outcomes in the first world ${y_{j, x}: j = 1, ..., N}$, is the same as the rank of $y_{i, x'}$ among all outcomes in the second world ${y_{j, x'}: j = 1, ..., N}$.

Then, for an illustrative example in clinical medicine, we might consider a scenario where each individual’s underlying health status ($U$) is fixed (here we don't consider $Z$ for simplicity), and thus its rank is determined prior to treatment. If all patients received the same treatments, then the rank of outcomes for each individual remain the same rank of the individual underlying health status.

W2:: There are some minor typographical and organizational issues that should be addressed.

Response to W2: Thank you for pointing this out. We will revise it accordingly. Thanks again.

Q1: The proposed loss function is a key component of the method. Could the authors provide more intuition and discussion behind its design?

Response to Q1: Thank you for raising this interesting question. Actually, the design of the loss function is more mathematical than intuition. Inspired by Koenker and Bassett (1978) [reference 32 of the manuscript], we found that $\frac{\partial \mathbb{E}[ |Y_{x'} - t| \mid Z=z ] }{ \partial t} = 2 \mathbb{P}(Y_{x'} \leq t \mid Z=z) - 1$ To construct a loss function whose first derivative satisfies $2 \\{ \mathbb{P}(Y_{x'} \leq t \mid Z=z) - \mathbb{P}(Y_{x} \leq y \mid Z=z) \\} = 0.$ It is natural to define the loss function as $R_{x'}(t| x, z, y) = \mathbb{E}[ |Y_{x'} - t | | Z=z ] + (1 - 2\mathbb{P}(Y_{x} \leq y \mid Z=z)) t = \mathbb{E}[ |Y_{x'} - t | | Z=z ] + \mathbb{E}[ \text{sign}(Y_x - y) | Z=z] \cdot t.$

Q2: The paper claims that the method can be extended to continuous treatments and that Proposition 5.3 remains valid in this setting. However, a formal proof for this claim is not provided. Could the authors please discuss on this point?

Response to Q2: Thanks for your comments. In Appendix D, we have presented the loss function for continuous treatments along with a brief proof. A similar derivation can also be found in Theorem 1 of [1]. We would be happy to provide a more detailed proof if the current presentation requires further clarification.

---

[1] Nathan Kallus and Angela Zhou (2018), Policy Evaluation and Optimization with Continuous Treatments, AISTATS

Thank you very much for your positive evaluation of our paper. Below, we hope that the our clarification addresses your concerns.

W1: I do not agree with line 169 stating arbitrary heteroskedastic models satisfy the condition.

Response to W1: Thank you for pointing this out. We fully agree with you and we will revise it accordingly.

W2: The authors should make the non-technical statement of their assumption explicit, rather than describing it in solely in terms of the rank coefficient.

Response to W2: Thank you for your helpful suggestions. Below, we provide more non-technical statements for Assumption 4.2 ($\rho(Y_x, Y_{x'}\mid Z)=1$).

Assumption 4.2 implies that an individual’s factual and counter factual outcomes have the same rank in the corresponding distributions of factual and counterfactual outcomes for all individuals. To better understand this, imagine two counterfactual worlds:

• In the first world, every individual receives treatment $x$;
• In the second world, every individual receives treatment $x'$.

For a given individual $i$, their outcomes in these two worlds are $y_{i, x}$ and $y_{i, x'}$, respectively. Assumption 4.2 states that the rank of $y_{i, x}$ among all outcomes in the first world ${y_{j, x}: j = 1, ..., N}$, is the same as the rank of $y_{i, x'}$ among all outcomes in the second world ${y_{j, x'}: j = 1, ..., N}$. In clinical medicine, we might consider a scenario where each individual’s health status ($U$) is fixed, and its rank is determined prior to treatment. Furthermore, this rank may influence—or depend on—the rank of potential outcomes after treatment.

W3: The authors did not benchmark their estimator in an example where "homogeneity + monotonicity" is satisfied. How would their estimation strategy would compare to the classical quantile-based estimator in this setting?

Response to W3: Thanks for your insightful comments. We add experiments by modifying the data generation process to make the data satisfying "homogeneity + monotonicity". Specifically, we set $\alpha = 1$ and add a constant in $Y_1$, i.e., $Y_1 = W_y · Z + U_1 + 1 \textup{ and } Y_0 = W_y · Z + U_1 \textup{ with } Wy \sim N (0, I_m).$ The corresponding numerical results are presented in the following tables. They show that under the 'homogeneity + monotonicity' setting, the proposed method performs less favorably than the quantile-based estimator.

Thank you very much for your positive evaluation of our paper. Below, we hope that the our clarification addresses your concerns.

Q1: The authors claim that Assumption 4.2 holds for common models such as those with additive noise $Y = g(X, Z) + U$ or heteroscedastic noise. Could you elaborate on why this is the case?

Response to Q1: Thanks for your comments and we apologize for any lack of clarity. Consider the additive noise model ($Y = g(X, Z) + U$) as an example. This model satisfies both the homogeneity and strict monotonicity conditions (Assumptions 3.1 and 3.2). Then by Proposition 4, $\rho(Y_x, Y_{x'}\mid Z)=1$ (Assumption 4.2) holds.

Q2: How can practitioners assess or empirically validate the rank preservation assumption in real-world applications?

Response to Q2: Thanks for your comments. In response to your question, we would like to clarify the following two points:

First, the rank preservation assumption is unverifiable empirically. This challenge is not unique---all causal inference methods rely on fundamental assumptions that cannot be directly tested, such as unconfoundedness or the validity of instrumental variables. The plausibility of these assumptions must instead be evaluated through domain expertise and contextual knowledge. Consequently, the choice of appropriate causal methods and identifiability assumptions depends on the specific problems at hand. This is the key difference between causal methods and non-causal methods.

Second, to assess the robustness of the proposed method, we typically perform sensitivity analyses under scenarios where key assumptions are violated. In Appendix E, we explored empirical performance with violated assumptions on rank preservation. It indicates that the proposed perform stably well even if the n rank preservation is violated.

Q3: In the experiments, the proposed method appears to outperform all competitors consistently. This is somewhat surprising, as methods usually have specific regimes where they excel and others where they do not. Could you provide some intuition for these results? Have you explored other scenarios or settings? Did you use simple methods for estimating nuisance functions for the baselines? This is particularly surprising in settings where the assumption is violated, such as in Table 4 of the appendix, where some competitive methods are missing. Also, in non-simulated settings, how do you choose or tune the bandwidth parameter?

Response to Q3: Thanks for your comments. We first would like to clarify that as shown in Table 3, although our method overall stably outperforms the baselines, the performance of CFQP on out sample $\sqrt{\epsilon_{PEHE}}$ is better than our method on the IHDP dataset, and the performance of CFRNet on in sample $\epsilon_{ATT}$ is better than our method on the Jobs dataset.

In addition, on the synthetic dataset, as shown in Table 1, our method outperforms among all scenarios, which is due to the data generation process already satisfying the ranking assumption. These results are intuitive: this is because all the baselines (except Quantile-Reg and Ours) are not designed for estimating counterfactual outcomes; they are designed to estimate the conditional average treatment effects $E[Y_{x'} \mid Z=z]$. In addition, they do not use the information of $y_{x}$ when estimating $y_{x'}$. For the estimation methods for counterfactual outcomes, incluing Quantile-Reg and Ours. Since the data-generating mechanisms violate the homogeneity assumption (Assumption 3.1), and the Quantile-Reg rely on the homogeneity assumption and thus underprior our methods.

Moreover, we explored other scenarios or settings as shown in Table 2 and Table 4. Taking Table 4 as an example, compared to the original results in Table 1, the advantage of our method is smaller, and some results (3 out of 8) are not significantly outperformed the best baselines.

We do not use simple methods for estimating nuisance functions for the baselines; all methods are implemented as it should be and all the results are compared in a fair way.

Finally, in non-simulated settings, for tuning the bandwidth parameter. We treat it as a hyperparameter and use grid search for selecting the bandwidth. We will add these details in the revised version. Thanks again.

Q4: The reliance on kernel-based estimation could limit scalability in high-dimensional or large-sample settings. How do you handle scenarios with many continuous confounders? In particular, in your experiments, the estimator appears quite stable even when the covariate dimension reaches 40; could you clarify why?

Response to Q4: We thank the reviewer for the questions. Note that we do not use kernel function directly on the original covariate. Instead, we first learn a low dimension covariate representation, then use this representation as the input, see the code in the supplementary for more details (line 42 in Ours1.py file). This is a standard implementation in the community of causal machine learning. We will clarify this issue in the revised manucript.

Q5: Finally, the abstract states that rank preservation is “not stronger” than homogeneity and strict monotonicity, whereas the main text (e.g., Proposition 4.1) shows that it is in fact strictly weaker. Clarifying this language in the abstract would help avoid potential confusion for readers.

Response to Q5: Thanks you for pointing this out. We will revise the abstract to make it consistent.

Q6: In line 120, the references cited do not seem directly relevant to the point made. Highlighting heterogeneity does not necessarily imply that one must estimate individual-level effects rather than conditional effects.

Response to Q6: Thank you for your valuable comment. The cited references here all emphasize the distinction between individualized treatment effects and conditional average treatment effects, rather than asserting that one must estimate individual-level effects instead of conditional effects. We will revise this subsection to clarify this point further.

Q2: Thank you for the responses, but unfortunately they do not fully satisfy me. In causal inference there are hypotheses that are not testable; however, we still try to provide guidance to users. For example, for the unconfoundedness assumption, by discussing with experts, you try to build a DAG and identify the variables related to the treatment and the outcome, and then, as you mention, you also turn to sensitivity analysis.

My point, which is in fact shared by the other reviewers, is that more intuition and guidance are needed on this key assumption. You have provided some interpretative elements in your responses to the other reviewers, and it is really crucial to include these in the main body of the text, while also trying to go further and provide concrete examples where the assumption makes sense.

The method works better under this assumption, and as shown in the new simulation, the results may be less favorable than the quantile method under the “homogeneity + monotonicity” assumption, which makes sense. But for a user working with data, what would you recommend, and in which cases? The paper would greatly benefit from an in-depth discussion of these scenarios to maximize its practical impact.

Q3: I am not sure you have answered to the question that was why in table 4 you do not have all the same methods than in table 1? Could you provide the results with all the methods? In addition, you answer:"We do not use simple methods for estimating nuisance functions for the baselines; all methods are implemented as it should be and all the results are compared in a fair way." Please, can you be more specific? I was asking how the nuisance components are estimated and to give more details on the estimation.

Q4: Thank you for the clarification, but the details should be included in the manuscript. Estimation is important and if you use dimensionality reduction before applying kernel, which indeed makes sense in high dimension it should be stated also to better understand the impact of the different choices.

Dear Reviewer 4AFD,

Thank you very much for your positive evaluation of our paper and for your helpful suggestions. Below, we hope that our clarification addresses your concerns.

Q2: My point, which is in fact shared by the other reviewers, is that more intuition and guidance are needed on this key assumption. You have provided some interpretative elements in your responses to the other reviewers, and it is really crucial to include these in the main body of the text, while also trying to go further and provide concrete examples where the assumption makes sense. The method works better under this assumption, and as shown in the new simulation, the results may be less favorable than the quantile method under the “homogeneity + monotonicity” assumption, which makes sense. But for a user working with data, what would you recommend, and in which cases? The paper would greatly benefit from an in-depth discussion of these scenarios to maximize its practical impact.

Response: Thank you very much for your kind comments and helpful suggestions. We fully agree with you that providing guidance on assumptions to users is important for real-world applications.

We apologize for not fully addressing the reviewer's concern in our previous response due to our misunderstanding, and we are glad that you noticed we provided some interpretative elements in our responses to the other reviewers. Following your suggestions, we added an additional example below to illustrate the rank preservation assumption (heterogeneity + monotonicity).

In addition, we provide more heuristic discussions.

First, as a direct response, compared with the quantile method (ref. 13 in the manuscript), we recommend our method. The reasons are two-fold:

• (a) Our method relies on weaker assumptions (Proposition 4.4) and accommodates "heterogeneity", making it applicable to a broader range of real-world scenarios;
• (b) Our empirical results show that our method outperforms the quantile method in the setting "heterogeneity+monotonicity", and performs slightly better or comparably to the quantile method in the setting "homogeneity + monotonicity".

Second, although our rank preservation assumption is weaker than the combination of "homogeneity + monotonicity" assumptions, it is still a strong assumption and may be violated in practice. Thus, in practice, we also recommend conducting a sensitivity analysis to further assess the robustness of the conclusions to potential violations of this assumption. Specifically, when the rank preservation is relaxed into the following Assumption R:

• Assumption R: $| P(Y_{x} \leq y_x \mid ) - P(Y_{x'} \leq y_{x'} \mid ) | \leq \alpha$ for $0 < \alpha < 1$. When $\alpha=0$, Assumption R reduces to the rank preservation assumption. Then we could explore how the bounds on $y_{x'}$ varies with $\alpha$ (a novel sensitivity analysis method). By theoretical analysis (similar to Theorem 5.2), we can show the following conclusion.
• Let $R_{x'}(t| x, z, y, \beta) = E[ |Y_{x'} - t | | Z=z ] + E[ \text{sign}(Y_x - y) | Z=z] \cdot t + 2 \beta t$ Under Assumption R, $y_{x'} \in [ y^l, y^u ]$, where the lower bound $y^l$ is the unique solution of $R_{x'}(t| x, z, y, \alpha)$, and the upper bound $y^u$ is the the unique solution of $R_{x'}(t| x, z, y, -\alpha)$.

Thus, we can conduct experiments to evaluate how the conclusions (or $y^l$ and $y^u$) vary with the value of $\alpha$ in real-world applications.

Thank you very much for your positive evaluation of our paper. Below, we hope that the our clarification addresses your concerns.

W1: All identifiability assumptions (including this) are restrictive. The paper would benefit from a more candid discussion acknowledging that all identifiability assumptions in counterfactual inference, including rank preservation, impose unverifiable structure on unobserved quantities. Comparing empirical consequences of assumption violations (as partially done in Appendix E) is a good direction that could be expanded.

Response to W1: Thanks for your helpful suggestions. We agree with you that all identifiability assumptions are restrictive and are unverifiable or untestable. We will add a discussion on it in the conclusion.

We also agree with you that comparing empirical consequences of assumption violations is helpful. In Appendix E, we explored empirical performance with violated assumptions on rank preservation. As a complement, we may further theoretically investigate the impact of the violation of rank preservation. Specifically, when the rank preservation is relaxed into the following Assumption R:

• Assumption R: $| \mathbb{P}(Y_{x} \leq y_x \mid Z=z ) - \mathbb{P}(Y_{x'} \leq y_{x'} \mid Z=z) | \leq \alpha$ for $0 \leq \alpha < 1$.

When $\alpha = 0$, Assumption R reduces to the rank preservation assumption. Then we could explore how the bounds on $y_{x'}$ varies with $\alpha$ (a novel sensitivity analysis method). By theoretical analysis (similar to Theorem 5.2), we can show the following conclusion.

• Proposition 5.7: Let $R_{x'}(t| x, z, y, \beta) = \mathbb{E}[ |Y_{x'} - t | | Z=z ] + \mathbb{E}[ \text{sign}(Y_x - y) | Z=z] \cdot t + 2 \beta t$ Under Assumption R, $y_{x'} \in [ y^l, y^u ]$, where the lower bound $y^l$ is the unique solution of $R_{x'}(t| x, z, y, \alpha)$, and the upper bound $y^u$ is the the unique solution of $R_{x'}(t| x, z, y, -\alpha)$.

W2: Reliance on propensity score estimation and kernel choices. More systematic sensitivity analyses, especially in high-dimensional settings, would clarify the method's reliability. Exploring alternatives to kernel smoothing for scalability would also improve practical value.

Response to W2: Thanks for your constructive suggestions. Empirically, we have explored several sensitivity analyses for various compoents of the proposed method, such as covariate dimensions (Tables 1 and 2), violation of rank preservation (Table 2 and Appendix E), heterogeneity degrees (Figure 1), and kernel choices (Appendix E).
Theoretically, we complement a sensitivity analysis for the violation of rank preservation (see our response to W1). For more discussion on the high-dimensional settings, we leave it to future work. Thanks again for your kindful comments.

Dear Reviewer niUF,

Thank you very much for your kind comments and timely feedback. In response, we have conducted additional sensitivity analyses in high-dimensional settings. The associated results are shown in Table below, where the data-generating processes are the same as those in Table 1 of the manuscript, except that the covariate dimension is increased from 5–40 to 80–160.

From this table, we can see that our method stably outperforms the competing methods.

We hope that the above clarification addresses your concerns.

Warm regards,

Authors