A Dynamical View of the Question of Why

We thank the reviewer for the feedback. Here, we address all the points.

I think you are being a bit unfair to the research community with your first sentence …

We covered the literature on both static and dynamic settings in the Related Work section (we will update that section in light of new pointers). What we meant by the opening sentence is that causation in dynamical settings are far less attended to in the existing literature. If this still sounds unfair, we will update it to clarify.

"If cause event A had not occurred, effect event B would not have occurred" this sentence does not make grammatical sense.

This sentence is both correct and accurate. Indeed, you can see this repeatedly in various philosophical texts when talking about counterfactual analysis. See for example Paul’s classic paper, 1st page right under “Lewis’s Analysis” (that’s why we have placed this statement inside quotation marks):

L. A. Paul, ‘Problems with Late Preemption’, Analysis, vol. 58, no. 1, pp. 48–53, 1998.

To help rectifying your mind, we italicise cause event A.

The reviewer can further refer to this work to better understand this fundamental philosophy of causation: Menzies, Peter and Helen Beebee, "Counterfactual Theories of Causation", The Stanford Encyclopedia of Philosophy (Winter 2020 Edition), Edward N. Zalta (ed.), URL = https://plato.stanford.edu/archives/win2020/entries/causation-counterfactual/

what are these limitations? Pearl has multiple works …

We would like to point out that the next few sentences of the paragraph highlight these with references. However, we will further discuss here some of the major differences along with the shortcomings of the interventionist/manipulationist framework:

1. Difficulty/impossibility of interventions in many dynamical settings: Unlike the interventions/manipulations approach to understand causation, we can make our conclusions from raw observational data without having to conduct interventions or worry about the kind of interventions, especially in complex systems. Further, interventions might not be possible in all physical systems, as we’ll discuss in detail in the answer to the next question.

2. Need to know causal graphs (a strong assumption) and scaling poorly with the number of variables. We discuss this in detail w.r.t the Atari example, in the response to another question of the reviewer.

3. Confusion about time of happening an event: our definition of an event clears all the confusion around the time of happening of an event and provides the flexibility to study multiple events that share the same ruling variables and admit the same changes but occur at different points in time.

4. Other interventions between events: Instead of considering actions under a fixed policy, i.e., restricting the chain of events between $A$ and $B$, we examine if event $A$ causes $B$ under the most pessimistic version of such chains of events. Remark that adhering to a certain chain of actions can also be readily considered in our framework.

5. These approaches ignore the dynamics as well as the possibility of other interventions between events (as explained later).

In order to get specific differences between our framework and the Halpern and Pearl framework for causation [Joseph Y Halpern and Judea Pearl. Causes and explanations: A structural-model approach. part ii: Explanations] please refer to the last section of Appendix C.3.

We appreciate the reviewer for the positive and encouraging feedback. We discuss the questions below. Please do let us know if the explanations are clear and whether you have more questions.

The general problem setup reminds me of what some in the field call "reverse causality".

Thanks for the pointer, we would add the citation. This seems to be a short paper that suggests forward and reverse questions. While not providing any actual formulation (and computational framework), the view is interesting and must be acknowledged.

Definition of cause/grit (medical event A -> C -> B).

In your example, if $C$ happens (or can happen) with some non-zero probability, even unknown or non-stationary, then $A$ is a cause of $B$ and is also correctly captured by our theory [the grit indeed increases by $A$ as shown below]. On the other hand, if $C$ is chosen not to happen at all, then $B$ will never happen. In this case, $A$ will not be considered a cause because we violate condition C1 (in our definition of causation), where $A$ must happen before $B$ - here $B$ never happens! We note that in the lack of event $B$, there is no causal question. Put it a bit differently, $A$ is a cause of $B$ conditioned on $C$ being selected with non-zero probability (and this is exactly what our framework deduces). That is, grit is meaningful only if $B$ is not unlikely to happen.

As a side note, we have also observed this computationally in the Diabetes example. W.r.t. insulin intakes, we observed that any insulin intake shows an increase in expected grit -- even when it is insufficient to cause event $B$ by itself and at least another sufficiently large intake is required. This is precisely what is expected from the theoretical analysis.

Analysis: First, please recall that the definition of causation asserts the expected change of grit must be positive from before to after $A$. Let $C$ happen with probability $p > 0$ and if it happens, the grit of $B$ increases by $\alpha > 0$ (since $A$ has already happened). Then $\mathbb{E}\Delta\Gamma_{B}(A) = p\times\alpha + (1-p) \times 0 > 0$. Hence, grit of $B$ cannot be the same from before to after $A$, and indeed it increases. Thus, $A$ is also a cause unless either $p$ or $\alpha$ is zero.

Why is the notion of grit the correct way

As we discussed above, the notion of grit works fully inline with what is expected intuitively as a causal relationship (even when it looks counterintuitive at the first glance). The minimum probability provides the natural way of dealing with possible future actions, as it ensures that $A$ is named a cause of $B$ only if it is so under any (even pessimistic) future trajectory. Importantly, if there is no action, then it automatically retrieves the standard definition (that $A$ should increase probability of $B$, rather than grit of $B$). Moreover, if a fixed policy is chosen to follow, either in general or only after event $A$, our framework still works readily, as $V^*$ will automatically be replaced by $V^{\pi}$.

No code is provided

We will release the complete code that reproduces all the results presented in the paper (and can easily be modified for other use-cases).

The notation of state $X(t)$ …

This is a common way, which basically only sets time as the base argument (mostly to parallel the discrete-time format). It’s being used in literature, see for example RL/DP papers in continuous time, such as this one:

Kim and Yang, ‘Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time’, in Conference on Learning for Dynamics and Control, 2020, vol. 120, pp. 739–748.

notation $V^{\pi}$

We fixed it; thanks for the pointer. $\pi$ needs to be in the condition of expectation.

We thank the reviewer for the feedback. We address each question/concern bellow. Please do let us know if these discussions are clear and satisfactory and whether you have more questions.

The reviewer find the paper very hard to read and understand.

We would like to highlight that this is a deeply interdisciplinary subject by nature and a certain level of complexity is unavoidable. This paper covers materials at the intersection of different communities (causality, systems control, reinforcement learning, and even philosophy). We tried to keep the presentation more formal and less descriptive with the hope that it helps the reader to follow the ideas in a precise manner, regardless of differences in community-related aspects. We indeed wrote several drafts of this work, and based on various internal feedback; the current version seems to be the most reliable as read by people with different backgrounds.

We, however, will do modifications before final submission to improve the presentation as much as possible.

Some of the assumptions are already well known in causality (“Causality necessitates time”. “ Cause happens before effect”).

We are respectfully not sure why this point is set as a weakness?

In any case, we would like to emphasize that these are NOT our assumptions, rather these are basic axioms which are well-known and are true in any causal arguments. We simply place them at the forefront. We will modify the text to clarify this point.

The reliance of the proposed method on optimal value function corresponding to reward function seems a strong one.

It is a fair point but it is also the case for any method that uses neural nets or approximation techniques (and we add a statement to reflect on this in the paper). Nonetheless, what is important here is that, using optimal value functions, our framework provides a strong basis (with formal grounds) to design a learning problem and gives all the computational tools to establish causal links in dynamical systems -- something which did not exist before, at least not at this level of formal rigour and explicitness, while still practically applicable and useful. Then, in practice, the optimal values will be replaced with the learned ones from data. As with ANY learning problem, the results will be reliable to the extent of approximation errors; this is not a specific problem of our framework, rather is a generic issue.

Of note, additional theoretical arguments can also be made here. For example, if the approximated value function is off by some maximum amount, then under certain assumptions, the degrade in validity of causal arguments will also be bounded (proof sketch: the bound over values will map to the same bound over grit, which will then be used in the decomposition lemma to bound the error in each component). Or, if the value of a certain policy is used in place of $V^{*}$, the causal arguments will be completely valid if that policy is followed after event $A$ (this result is immediate from the definition of value functions).

We would like to remark that presenting such results can go much beyond the scope of a single conference-paper.

will be interesting to investigate how the proposed method works as the function of number of variables.

Our framework scales well to even huge state-spaces simply because it maps the problem into RL settings (and current deep RL methods can easily handle extremely large state-spaces). Even in the provided Atari problem, the state comprises 28224 variables (corresponding to the pixels of four consecutive game screens)! The T1 diabetes simulator, on the other hand, deals with 13 state variables and 2 action variables. This reflects the range of variables our proposed method can handle.

In a related context, as the base problem here is an RL problem (due to our Value Lemma), any value-based RL technique can be used, and the scalability will be inherited from the properties of the RL algorithm.

We thank the reviewer for the very encouraging feedback. Here we address the questions; please let us know if there are further questions/clarifications.

structure of the paper

We do appreciate the suggestion (and indeed liked it). However, the page limit does not allow for such a presentation, unless a significant portion is omitted. As this paper covers materials at the intersection of different core topics (causality, systems control, reinforcement learning, and even philosophy), we believe it would be required to cover them all and there is little room to shorten the paper. We tried to keep the presentation more formal and less descriptive/intuitive with the hope that it helps the reader to follow the ideas in a precise manner, rather than motivating at the forefront. We indeed wrote several drafts of this work, and based on various internal feedback; the current version seems to be the most reliable as read by people with different backgrounds. We, however, will do modifications before final submission to improve the presentation as much as possible.

what is HP

It stands for Halpern and Pearl. The explanatory sentence should unfortunately have been deleted in one of the last revisions. Will fix it.

MDP

We will add a clarification why MDP is defined (when it is introduced) to help address your point.

Does this framework need the ability to take actions (i.e. do we need an MDP), or is that just an estimation technique?

Our framework does not need the ability to take actions, and you certainly can use this theory for a system with only states (no action).

The reason to use MDP is basically the other way round: a system/environment may include actions and the framework needs to be able to cover such cases as well; hence, MDP is needed in general. Moreover, as we discussed in the paper, an event may also be defined as taking an action (a change in an action variable) rather than a change in a state variable. Our theory covers such cases too, which is again why we need MDP in general.

I just observe a dynamical system but don't have the ability to take actions, I can't infer causality in this framework, right?

You definitely can. This case can happen if either there is no action in the system by nature, or a fixed policy is taking care of actions with no freedom (hence, there is effectively no action). In all state-only problems, you simply use $\Gamma_{B}(X)$ and $V_{\Gamma}(X)$ and the rest of the theory remains valid and intact.