Detecting and Measuring Confounding Using Causal Mechanism Shifts

When the environment changes, for example, if c changes to c', will the original causal relationship change?

It depends on the type of intervention. If context change is a result of soft intervention on a variable, the underlying causal relationships do not change. If the context change is a result of hard intervention, the causal relationships can change. A similar discussion can be found in Section 2 of [35]. In this paper, we consider both types of interventions on variables as highlighted in Table 1.

If the length of the environment node is only 1, will the method described in the article fail?

When there is only one context available, there is a fundamental limitation: confounding cannot be uniquely identified, as explained in lines 43-45. Rather than calling it a limitation, we state that our method is not applicable when there is only one environment.

Why do settings 2 and 3 introduce mutual information to define confounding? Would there be any difficulties ..?

We build upon earlier works that utilize mutual information to measure confounding. According to our formulation, mutual information is among the most suitable choices for measuring the dependency between random variables induced by changes in the mechanism, as explained in lines 295-297. See [35] for a similar approach using mutual information for identifying confounding. In our experiments, we did not encounter any difficulties using KL divergence.

In line 104, the author said "an approach we also leverage in this work", but the author did not introduce the method in which article in detail.

Apologies for the confusion. By "an approach," we refer to the idea of using different contexts to measure confounding. We will either modify or remove the text in brackets to clarify this point and avoid any confusion.

In line 111, the article describes "we measure the effects of both observed and unobserved confounding". However, the subsequent...

We believe the reason for confusion is the word 'effect'. We aim to study and quantify the confounding bias instead of its effect on causal effects as a downstream application. Our goal in this paper is to provide a unified framework for confoudning. We will replace effect with bias to remove the ambiguity.

We appreciate your interest in the results on causal effects. We conducted experiments to study the impact of the proposed confounding measures on reducing bias in estimated causal effects. Table 2 in the uploaded rebuttal PDF demonstrates that controlling for the variables detected as confounders by our method helps reduce the bias in estimated causal effects.

In fact, the confounding effect is not always a number between 0 and 1. Converting it to a number between 0 and 1...

We provide a reason for studying relative strength in lines 57-62 of the introduction section. Also, it is trivial to evaluate the actual confounding effect that is not between 0 and 1 by ignoring the exponential transformation used in our definition of confounding measure.

In Setting 1: When there is confounding between two variables, CNF-1 is used to calculate it. If multiple variables have common confounding...

Since this is a first effort to define a measure of confounding among a set of variables, our focus herein was on the stated settings. It is possible however to accumulate the confounding. It is easy to see that our current definition of measuring joint confounding satisfies positivity and monotonicity properties. We believe that exploring other ways of defining joint confounding can be interesting directions of future work.

The detect confounding method proposed in the article, but this measure is relatively rarely ...

We included two simple real-world applications of our method in the uploaded rebuttal PDF, as shown in Figure 1. (The purpose of these examples is only to show its practical value, and not intended to be comprehensive or complex.) For the conditional confounding experiments, we observe similar results for Settings 2 and 3; therefore, we report results for Setting 2 only. Additionally, our experiments on conditional confounding assume that the confounding variables are observed. We have provided additional experimental results in the uploaded rebuttal PDF, detailed in Tables 1 and 2, which demonstrate the usefulness of our methods.

The authors miswrite $D^c$ as $D$ in the third line of ALgorithm 1.

As explained in line 245, we combine data from all contexts to evaluate conditional probability. We will update third line of Algorithm 1 to include $\mathcal{D} = \cup_{c}\{\mathcal{D}^c\}$ to make it clear.

References: [35] Mameche, Sarah, Jilles Vreeken, and David Kaltenpoth. "Identifying Confounding from Causal Mechanism Shifts." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

We hope our responses address your concerns. We will update the manusript accordingly. We are happy to discuss further if you have any additional questions.

I found the paper to be lacking in experimental evaluations. It is not clear how hard (both statistically and computationally) it is to compute the measures of confounding that were proposed, especially the ones measuring confounding among multiple variables.

Due to space constraints, we included experiments in Appendix Section B (our paper required significant space for discussing the considered 3 settings and our formulations therein). We have now included additional experimental results in the uploaded rebuttal PDF. If accepted, we will use the additional page available in the final version to include the results in the main paper itself.

As stated in the appendix, our experiments were conducted on a CPU and are straightforward to run. We have provided the code in the supplementary material, and it can be executed quickly.

I am somehow skeptic about the practical relevance of the results presented in the paper ... people focused a lot on the marginal sensitivity model ... I am not sure the same would be possible under the proposed measure of confounding, and hence I am not sure how it would be useful in practice. I think some relevant related works are missing...

In the uploaded rebuttal PDF, Figure 1 demonstrates two simple real-world examples where our methods can be applied (the purpose of these examples is only to show its practical value, and not intended to be comprehensive or complex). As explained in the previous response, we have included experiments in the Appendix and new experiments in the uploaded rebuttal PDF.

We appreciate your suggestion regarding sensitivity analysis. We will incorporate the following points in the revised manuscript for completeness.

1. Our method aims to estimate the exact value of confounding rather than approximate it using bounds obtained via sensitivity analysis.
2. The difference between the total confounding and the conditional confounding by conditionining on observed confounding variables can be interpreted as the unobserved confounding estimate obtained via sensitivity analysis.
3. As discussed with reviewer n1WD, we can connect marginal sensitivity definition of confounding and the confounding based on directed information used in our paper. We will include this connection in the revised manuscript.

We also thank you for providing the relevant papers. We will discuss them in the revised manuscript and include a discussion on sensitivity analysis as mentioned before for completeness.

Q1) I got confused at lines 103-106: "While these methods use data from different contexts... [27] proposes a test for the presence of hidden confounding and hence allows for unobserved variables... compare with the approach proposed in [27]?

Apologies for the oversight. The study of hidden confounding detection in [27] primarily focuses on downstream causal effect estimation. In contrast, our work aims to provide a unified framework for studying and measuring both observed and unobserved confounding across different types of contextual information. This framework supports various downstream applications beyond causal effect identification, including assessing the relative strengths of confounding and measuring confounding between pairs and sets of variables. We will add these points to the related work in the revised manuscript. We also study downstream causal effect estimation tasks. As demonstrated in the results in the rebuttal PDF, confounding detection using our method contributes to improved causal effect estimation.

Q2) I am slightly confused with Assumption 3.2 (line 169): what does it mean for conditional distributions to be different?

Consider a variable $X_i$ and its parents $PA_i$. If $\mathbb{P}(X_i=x_i|PA_i = pa_i)$ is different in two contexts/environments $c,c'$ for atleast one pair $(x_i, pa_i)$, we say that the causal mechanisms are different in two contexts $c,c'$. According to Assumption 3.2, such causal mechanism shifts are rare/sparse for a variable $X_i$.

(Q3) Can the authors comment with one motivating example of how their proposed confounding measure could be used in practice? (See W2)

We have included two simple real-world examples in the uploaded rebuttal PDF (Figure 1) demonstrating the applicability of our methods. Additionally, since the benchmark datasets from the BNLearn repository come with known data-generating processes, we can generate different contexts by performing interventions and subsequently applying our method, similar to [35,37].

References: [27] Rickard Karlsson and Jesse Krijthe. Detecting hidden confounding in observational data using multiple environments. Advances in Neural Information Processing Systems, 36, 2023. [35] Mameche, Sarah, Jilles Vreeken, and David Kaltenpoth. "Identifying Confounding from Causal Mechanism Shifts." International Conference on Artificial Intelligence and Statistics. PMLR, 2024. [37] Mooij, Joris M., Sara Magliacane, and Tom Claassen. "Joint causal inference from multiple contexts." Journal of machine learning research 21.99 (2020): 1-108.

We hope our responses address your concerns. We will update the manusript accordingly. We are happy to discuss further if you have any additional questions.

On line 158 and 169: p is defined as the inequality between two distributions, this definition doesn’t make sense... why overloading it?

We undestand the concern. To fix this, following [35], we will edit line 158 in the revised manuscript as follows.

"For example, the $p\text{-value}(\mathbb{P}^c(X_i|\mathbf{PA}_i^o)\neq\mathbb{P}^{c'}(X_i|\mathbf{PA}_i^o))$ where $\mathbf{PA}_i^o$ is the set of..."

On definition 4.1 what they call directed information is not an -expected- KL divergence, it is simply a KL divergence.

We agree with you. We will change "expected KL divergence" to "conditional KL-divergence" in the revised manuscript.

On Table 2, on rows 1 and 2 the equality with 0 is inverted...

We beg to differ. If $X_i\rightarrow X_j$, then we have $\mathbb{P}(X_i|X_j) \neq \mathbb{P}(X_i|do(X_j))$ and hence $I(X_i\rightarrow X_j)>0$. This is because, in $X_i\rightarrow X_j$, conditioning on $X_j$ does not make $X_i, X_j$ independent. However, intervening on $X_j$ makes $X_i, X_j$ independent when there is no confounding between $X_i, X_j$. Formally, when there is no confounding, $\mathbb{P}(X_i|do(X_j)) = \mathbb{P}(X_i)$. On the other hand, $\mathbb{P}(X_i|X_j)\neq \mathbb{P}(X_i)$.

Is definition 4.6 really a definition? That sounds like a proposition to me.

We agree with you, we will change the definition to proposition. The proof is trivial and follows from the definition of mutual information, which we will include in the revised manuscript.

I would like to get a feeling for why the properties of their measure of confoundedness are valuable...

We believe that these properties are essential to ensure the correctness of the proposed metrics from a measurement perspective. The monotonicity property is crucial for understanding which set of variables are more confounded than others. Since our metrics are derived from positive quantities like directed information and mutual information, they return positive values. Studying negative confounding is a potential future direction.

The authors do include a couple of sentences at the end of the paper where they state... I think it is too strong of a limitation.

We'd like to point out that we only didn't focus on real-world application, and rather deferred it to future work to apply our methods to real-world datasets. We have included two simple real-world examples in the uploaded rebuttal PDF (Figure 1) demonstrating the applicability of our methods (the purpose of these examples is only to show its practical value, and not intended to be comprehensive or complex). For example, we can study settings 2 and 3 using the SACHS dataset [35,37]. To study setting 1, it is enough to check for datasets that are obtained from randomized control trials where a set of variables are intervened.

We will modify the line in conclusions to reflect this point.

References: [35] Mameche, Sarah, Jilles Vreeken, and David Kaltenpoth. "Identifying Confounding from Causal Mechanism Shifts." International Conference on Artificial Intelligence and Statistics. PMLR, 2024. [37] Mooij, Joris M., Sara Magliacane, and Tom Claassen. "Joint causal inference from multiple contexts." Journal of machine learning research 21.99 (2020): 1-108.

We hope our responses address your concerns. We will update the manusript accordingly. We are happy to discuss further if you have any additional questions.

The assumption "that the causal mechanism changes are known for each variable across different contexts" ..

To measure confounding between all pairs of nodes in a causal graph, we need to know the mechanism changes for each variable across contexts. However, if the number of nodes among which confounding is to be identified is small, we only need to know that the causal mechanisms are changed for that small set of nodes, but not for all nodes. This assumption has been made in [35, 37]. We will edit line 160 as follows to make it clear that the assumption is not too strong.

"Hence, we focus on detecting and measuring confounding among a set of variables, assuming that the causal mechanism shifts are observed among that subset of variables."

The formal definition of a mechanism shift..?

We thank you for the suggestions. We will ensure that the intuitive explanations of mechanism shifts appear early in the paper. In particular, we will move lines 131-133 and lines 163-169 to introduction line 48 to introduce causal mechanisms and how the shift in causal mechanisms are useful in detecting confounding. Also, it is sufficient to know relative shift between environments, where each environment is a result of either a soft or hard intervention to a set of variables.

The relationship between definitions 4.2 and 4.3 and ignorability....

This is an interesting insight. We thank you for bringing up this point. The above paper presents an intriguing way of defining confounding as the difference between nominal and complete propensity scores in the potential outcomes framework. This is inherently connected to the ignorability assumption. Since directed information also relies on the KL-divergence between conditional and interventional distributions, we can leverage this definition to define relative confounding strength between sets of variables. We will include all of these points in the revised manuscript and discuss them in the related work section.

While the theoretical results are thorough...

Our primary focus in this work was on effectively identifying confounding variables, which is non-trivial by itself. Once we effectively identify observed confounding variables, we can control for them to reduce bias in estimated causal effects. We appreciate your interest in the results on bias in estimated causal effects. We conducted experiments to study the impact of the proposed confounding measures on reducing bias. Table 2 in the uploaded rebuttal PDF demonstrates that controlling for the variables detected as confounders by our method helps reduce the bias in estimated causal effects.

Definition 3.1 Looks like X is being redefined here ...

Definition 3.1: We undestand the concern. We will use a different variable in Defintion 3.1, say $\mathbf{W}$, in place of $\mathbf{X}$. With this change, none of the other contents gets affected.

Line 131: We will edit line 131 as follows.

"For a node $X_i$, $\mathbb{P}(X_i|\mathbf{PA}_i)$ is called the causal mechanism of $X_i$."

Assumption 3.1 talks about changes in causal mechanisms, but this concept hasn't been formalized...

Assumption 3.1 discusses independent causal mechanisms but not changes in causal mechanisms. We introduce the concept of causal mechanism shifts in lines 145-146 and later in Assumption 3.2, we discuss sparse causal mechanism shifts.

Line 147: what is a context node?...

Thank you for your suggestions. A context node can be viewed as an exogenous parent to the set of nodes on which an intervention is performed. Our analysis does not depend on this extended causal graph. However, we will define context specific distributions as you suggested to make this clear.

Definition 4.1: The notation is a little awkward with the condition on $\mathbb{P}(X_{j})$ within $D_{KL}$.

We followed [60] to use this particular notation of conditioning on $\mathbb{P}(X_{j})$ within $D_{KL}$. We believe this is to differentiate between conditional and interventional probabilities. Since it is clear from the context, we will exclude conditioning on $\mathbb{P}(X_{j})$ in the revised manuscript.

Line 128: unclear notation, what information is $P_{x}(V), X\subseteq V$ adding here?...

We use $P_x (V), X \subseteq V$ and $P_*$ to formally define causal Bayesian networks. They do not have any relation to our method. $X$ should be bold here. We will fix this typo.

Line 154: "Let $C_{S\wedge \neg R}$ be the set of contexts in which we observe mechanism changes for the set of variables $X_S$ but not for the variables $X_R$." Changes relative to what?...

In your question, if $i\in S$, we have $P^c (X_i|\mathbf{PA}_i) \neq P^{c'}(X_i|\mathbf{PA}_i)$. If $i \in R$, the inequality becomes equality. The shifts do not have to be the same across all contexts. Contexts are a result of either soft or hard intervention on the nodes. Hence, the changes between contexts are relative to interventional distributions.

Line 158: This does not look like a p-value. This looks like an indicator taking a value of true or false, not a probability... Same questions for Assumption 3.2.

Following [35], we will edit line 158 as follows:

"For example, the $p\text{-value}(\mathbb{P}^c(X_i|\mathbf{PA}_i^o)\neq\mathbb{P}^{c'}(X_i|\mathbf{PA}_i^o))$ where $\mathbf{PA}_i^o$ is the set of..."

The above p-value tests whether the causal mechanisms of $X_i$ are different in two different contexts $c,c'$. We will remove $p$ from line 158 to avoid confusion. In assumption 3.2, similar to [35], we use $p$ to indicate the probability of two distributions being different.

Line 286: Are the $\epsilon$ normally distributed? or is zero mean and any distribution ok?

The only restriction on $\epsilon$ is that it has zero mean with no other restriction on the underlying probability distribution.

Definition 4.7: Lower case $c$ is a bit confusing. We will replace $E^c_i$ to $E^C_i$ to avoid confusion in the revised manuscript.

Thank you for replying to my review. After reading your responses, I have decided to raise my score to a 5. I still think this paper needs significant revision. I choose to trust that the changes promised to all reviewers will be made, and leave the final decision to the AC.