Towards Characterizing Domain Counterfactuals for Invertible Latent Causal Models

We thank the reviewer for the constructive feedback. We respond to the questions and suggestions individually below. We have also modified the paper accordingly.

“The simulated experiments look very contrived to me. The data generation mechanism used for creating data in these experiments is too simplistic.”

We apologize for any lack of clarity regarding the data-generating process in the main paper, and we have revised the simulated experiment section to clarify the form of the observation function (marked in blue). To address the concern about the simplicity of our simulated experiments, we have added additional experiments where we use a more complicated normalizing flow (based on RealNVP) for the observation function in the ground truth dataset, and our model is misspecified as either a different normalizing flow structure or a VAE structure. More details and relevant figures can be found in Appendix J.2. In summary, we observe a similar trend as that in Figure 1, using the ILD-Relax-Can model results in lower counterfactual error than the ILD-Dense model, regardless of whether the structures of $g^*$ and $g$ match.

Motivation of the simulated experiment.

The primary aim of our experimental section is to demonstrate the practical utility of our theoretical framework. We begin with scenarios where all our assumptions are met, then progressively explore cases where these assumptions are not fully satisfied, such as model misspecifications in the intervention size. Following the reviewer's suggestion, we have further extended our investigation to include scenarios with misspecified functional classes of $g$. Finally, in Section 5.2, we test our model design in a more practical setting where we relax the invertibility assumption.

We also wish to highlight that our simulated experiments encompass a variety of distinct latent causal mechanisms. We have investigated 10 different SCMs for all of our experiments as discussed in Appendix G.1.

“However, I can't find an explanation of how sparsity is enforced in these experiments. In summary, I think authors can restructure the paper to provide more room for experiments in 5.2., and explain the methodology in detail.”

In Section 5.2, sparsity is enforced by using the sparse $f_\cdot$ structure (i.e. only $k$ mechanisms in $f_d$ are allowed to change across different domains) as the final layer of the VAE encoder and the initial layer of the VAE decoder. Informally, this can be thought of as the encoder of the VAE generating the Gaussian parameters in a sparse fashion and the decoder of the VAE is sparse due to its initial layer being sparse. An overview of this VAE setup can be seen in the encoder/decoder graphics in Figure 17.

“I think the writing and structure of the paper can be improved. A large fraction of space is used for many theorems, some of which are not that important. I suggest moving these theorems to the appendix and only keeping Theorem 2 in section 3.2.”

Thank you for your feedback. We agree that the writing of the theoretical sections can be simplified and have substantially revised Section 2 accordingly. Regarding the importance of Theorem 1 and 4, we summarize how they play critical roles in our paper below.

Theorem 1: This theorem establishes both necessary and sufficient conditions for counterfactual equivalence. It forms the foundational basis for subsequent theorems in our paper as it allows for easy verification of counterfactual equivalence in ILDs and can facilitate the rapid construction of counterfactual equivalence models for any given ILD.

Theorem 4: Together with Corollary 5, Theorem 4 demonstrates that all counterfactually equivalent ILDs share the same sparsity in intervention. This insight is crucial as it shows that constraints on the intervention set size do not exclude any equivalent ILDs, and it significantly narrows the functional space in practical applications, which leads to our model design in practice.

We thank the reviewer for the insightful review. We respond to the questions and suggestions individually below. We have also modified the paper accordingly (highlighted in blue).

The idea of “intervention” and “counterfactual” in this paper is nonstandard and confusing … in particular, the “counterfactual” does not consider the past values of exogenous variables which are the gist of “counterfactual”.

Thanks for the question.

Intervention: We would like to first clarify the intervention considered in this paper. Here, we mainly consider soft intervention which corresponds to the change of functions in the structural assignment [1] [2]. We note this is different than a perfect intervention (e.g., a do intervention), as this does not completely eliminate the causal effect of the parents.

Counterfactual: In this work, we are specifically interested domain counterfactual queries (i.e. what would $\mathbf{x}$ look like if it had come from domain $d’$ instead of domain $d$?), which are a specific type of the general counterfactual query. Similar to the general counterfactual query, a domain counterfactual query exactly follows the three steps: abduction, action, and prediction [3]. For example, given an image $\mathbf{x}$ from domain $d$, we want to answer the question what this image would look like if it is from domain $d’$. We first use the inverse of our observation function $g$ and our latent SCM for domain $d$ to recover the exogenous noise $\epsilon$ given the evidence (abduction) as follows: $\epsilon = f^{-1}\_{d}\circ g^{-1}(\mathbf{x})$. Then, we perform the domain intervention (action) by exchanging the original $f_{d}$ with $f\_{d’}$. Finally, we use the recovered noise and intervened SCM to predict the counterfactual $\mathbf{x}\_{d\rightarrow d’}$ (prediction). Together this approach follows: $\mathbf{x}\_{d \rightarrow d’} = g\circ f\_{d’}\circ f^{-1}\_{d}\circ g^{-1}(\mathbf{x})$. Additionally, in our empirical analysis, we pay particular attention to the variables that should remain constant during counterfactual estimation. For instance, in our ColorRMNIST experiment, a key focus is on evaluating whether the color attribute is preserved when we alter the rotation aspect.

[1] Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., & Bengio, Y. (2021). Toward causal representation learning. Proceedings of the IEEE, 109(5), 612-634.

[2] Peters, J., Janzing, D., & Schölkopf, B. (2017). Elements of causal inference: foundations and learning algorithms (p. 288). The MIT Press.

[3] Pearl J Glymour M Jewell NP. Causal Inference in Statistics : A Primer. Chichester West Sussex UK: John Wiley & Sons; 2016. , p96.

The “generalized causal mechanisms” in Prop 2 are only a subclass of “invertible SCMs” … missing any piece of information might render the subsystems non-invertible.

Thanks for pointing this out. We agree that if we only assume the invertibility of $f$, we could not guarantee that the subsystem of $\widehat{f}^{(i)}$ is invertible to $\epsilon_i$ given only $z_{<i}$. However, without loss of generality, we could assume the partial order in $z$ such that parent nodes have smaller indices, then the overall function $f$ is invertible autoregressive. We believe the confusion is that we define the invertible SCM on the subfunction $\widehat{f}$, rather than on $f$ directly.

The claim on the generality of the model class (Prop 1) seems problematic…Thus, we need a Rosenblatt transformation for each sample point, and we cannot construct the F_p and F_q for the whole distributions p(X) and q(\epsilon).

We might be misunderstanding this concern, please forgive any misinterpretation if that is the case. The Rosenblatt transformation is constructed based on a joint distribution, not a single fixed point [1]. But perhaps the concern stems from the fact that in prop 1, $\mathbf{x}$ represents a random vector rather than a specific instantiation. It is under this case, where $\mathbf{x}$) is a random vector, that the Rosenblatt transformation is well-defined. Additionally, we note that the conditional CDFs are invertible w.r.t. their first argument given previous values $x_{<j}$, but the whole function is invertible because it is autoregressive. Intuitively, this means that you can recover the first variable using the 1D invertible CDF, and then recover the 2nd variable given the first recovered value, etc. Thus, we believe our original proof is correct. Does this answer your concern (perhaps we misunderstood)?

[1] Rosenblatt, Murray. “Remarks on a Multivariate Transformation.” The Annals of Mathematical Statistics, vol. 23, no. 3, 1952, pp. 470–72. JSTOR, http://www.jstor.org/stable/2236692. Accessed 17 Nov. 2023.

We appreciate the insightful feedback provided by the reviewer. We responded to the questions and suggestions you made individually below, with pointers to locations in the manuscript where the corresponding changes have been made (highlighted in blue).

Suppose $g = Id$, for some domains $f_d = Id$ and for others $f_d = -Id$. For a given counterfactual query, how can your method know whether to predict according to $x_{d’}$ or according to $x_{d’}=-x_d$? Is there an (implicit?) assumption somewhere to rule out cases like this?

Thank you for this insightful question. Our work does not rely on unstated assumptions, and your example pinpoints a crucial aspect of our theoretical framework. Our paper's theoretical contribution lies in characterizing the relationship between counterfactually and distributionally equivalent Invertible Latent Domain (ILD) models. We demonstrate that all counterfactually and distributionally equivalent ILDs share a common feature: the size of the intervention set. In the scenario you described, it is indeed impossible to infer the counterfactual query directly from the interventional data. Based on our distribution equivalence definition (Definition 2), in this case, the intervention set size is $m$ because all the variables are intervened on. By fitting the domain distributions, the counterfactual error (defined in the new Section 3.4 of our revised paper) is bounded by a distribution fit term and a worst-case error. In your example, because the intervention set size is $m$, we will incur a worst-case counterfactual error of $O(\sqrt{m})$. However, if we assume the sparse mechanism shift (SMS) hypothesis about the true model, then we can reduce this worst-case counterfactual error to $O(\sqrt{k})$ because only $k$ dimensions can be the negative of the original.

We expect that future work will be able to more fully explore and analyze counterfactual risk to improve domain counterfactual estimation building upon our results in this paper.

[In the] abstract, "all non-intervened variables have non-intervened ancestors": I suggest "... have no intervened ancestors".

Thanks for the suggestion. We rewrote this part in the revised manuscript marked in blue.

"Given our assumption ... distribution equivalence": This is apparently without assumptions (1) and (2), which were stated in the preceding sentence. Please make clearer in the text that you're now considering assuming only (3).

This is based on Assumption 3. We have revised the paper accordingly.

In corollary 3, "Prop. 8" should refer to definition 8.

We have revised the paper accordingly.

Section 3.3: "establish on" should be something else…In corollary 5, no requirement of continuity is currently stated, but I think it is needed.

We have edited these sections to be more clear (e.g., we added a constraint that $(g, f_d)$ for all $d$ are continuous in Corollary 5).

Hi Reviewer 4MQs, thank you for your helpful feedback. Have we adequately addressed your questions and concerns?

We thank the reviewer for the helpful feedback. We responded to the questions and suggestions you made individually below, with pointers to locations in the manuscript where the corresponding changes have been made (highlighted in blue).

“I would appreciate a more intuitive explanation of Theorem 1 and its implications.”

Thanks for the question. The significance of theorem 1 is to formally characterize the set of model’s which are counterfactually equivalent to each other. Intuitively, this means that any model within this set produces exactly the same counterfactual output (i.e. $x_{d_1 \rightarrow d_2}$), despite different underlying causal mechanisms, such as variations in recovered exogenous noise, observation functions, or implied distributions (e.g., it is possible to be counterfactually equivalent even though not distributionally equivalent, see a section discussing the relationship between distribution equivalence and counterfactual equivalence in the Response to All Reviewers.).

Key implications of Theorem 1 include:

1. This theorem reveals that, surprisingly, despite the true ILD, vastly different ILDs could generate equivalent counterfactuals even if they might not be distributionally equivalent.
2. This theorem proves that recovering the original causal model, which is a much more challenging task, is unnecessary for estimating domain counterfactuals.
3. It provides a framework to test whether two Invariant Latent Domain (ILD) models are counterfactual equivalents. It also facilitates the development of new counterfactual equivalent ILDs. Utilizing this construction ability, the proof of Theorem 2, Proposition 3, Theorem 4 and Proposition 5 all directly use this theorem.

“How should we precisely define the size of the intervention set, denoted as 'k'? …there is usually a reference to a latent causal model where no interventions have occurred. How do we accurately establish this reference latent causal model?”

Thank you for your insightful question. To clarify, we categorize a node as 'intervened' when there's a variation in its causal mechanism across different domains in our ILD model. This determination can be made using any domain distribution as a reference point if the intervention is soft [1]. For simplicity and without any loss of generality, we often assume domain 1 as this reference. In our revised manuscript, this concept is elaborated in Equation 9 under Definition 9, where the second equality reflects this reference-based perspective.

Alternatively, we can assess intervention by comparing pairwise causal mechanisms. This means that a latent node $j$ is considered intervened if the functions $\widetilde{f}\_{d}^{(j)}$ and $\widetilde{f}_{d’}^{(j)}$ differ for any domain pair $d,d’$. This approach is represented by the first equality in Equation 9.

[1] Kocaoglu, Murat, et al. "Characterization and learning of causal graphs with latent variables from soft interventions." Advances in Neural Information Processing Systems 32 (2019).

Hi Reviewer ji9w, thank you for your helpful feedback. Have we adequately addressed your questions and concerns?

Thank you for your response. Succinctly, we need to know $k^*$ for better counterfactual estimation if we want to achieve counterfactual equivalence by fitting the observational data distribution.

Theorem 4 suggests that any distributional and counterfactual equivalent models share the same intervention set size $k$. To aid in selecting $k$ in practice, we have introduced Theorem 6 which shows that misspecifying of $k$ will lead to a bias-variance tradeoff scenario, where if the estimated $k$ is larger than $k^*$, there will be a large variance on the estimation. However, if the estimated $k$ is smaller than $k^*$, you will not be able to find a distributionally equivalent model, which leads to a biased estimation. For a detailed explanation regarding the impact of misspecifying $k$ and how to choose it in practice, we refer the reviewer to the Choosing intervention set size $k$ and new bound paragraph in our response to all reviewers. Of all of our assumptions, the sparse mechanism shift assumption plays an important role here as it, together with our new Theorem 6, guides us to choose a relatively smaller $k$ unless we fail to achieve distribution equivalence, which is justified by our empirical study.