Causally Aligned Curriculum Learning

Weakness 2:

“Acknowledging limitations and situating their work in presence of related work

1. I saw that the related works section was relegated to the Appendix. There should at least be a pointer in the main paper to the related works. I found it helpful to read it, to understanding the positioning of this work, and think that it should be included in the main paper. The paper provides almost too many details about the method in the main paper, and at places is rigorous to the point of being pedantic. I think that the space would be better spent on situating the work properly.
2. The inputs/outputs of the authors' method is not presented in plain language. As I understand it, the authors' method assumes access to the SCM of the task, and a curriculum generator such as GoalGAN. Such details should be made much clearer, perhaps through a summary methods figure or by adding a discussion of limitations.”

2.1 Thank you for the suggestion; we have added a pointer in the introduction section to the related work. We have discussed how our work is related to the literature in the introduction section, a condensed version of the related work section in the appendix. Given the theoretical nature of the work, we strive for a precise and self-contained presentation of the results. Due to space limitations, we only kept the most closely related literature for positing and motivating our work in the introduction section. We have provided an extended version of the related work in the appendix G.

2.2 The plain text descriptions of input/output of algorithms 1 and 3 can be found in the paragraph beside the algorithm pseudo-code, which is also shown in each pseudo-code box's input/output field. We would also like to clarify that SCMs are not assumed as inputs. Instead, we only require causal diagrams as inputs, a graphical representation of the dependency relationships among the variables without any detailed causal mechanisms or transition probabilities (the F and P(u) in the definition of SCM in the preliminary section).

Weakness 3:

“Figures are not standalone: it is common practice to read a paper by skimming the figures + captions first, but I found that it was not possible to get the main idea of the figure, method and experimental results by doing so. Even after reading most of the accompanying text, I still found the figures + captions alone ambiguous. I had to find the specific part of the text referencing the figures -- sometimes needing to jump to other parts of the paper-- and read very carefully to understand what was happening. The authors should rewrite the figure captions and perhaps add a new "methods" figure summarizing the flow of their method. My specific comments on two figures are below, but all figures can be improved:

1. Figure 5: Specifically, I was confused about why the curves labelled "causal" performed differently across the four columns? Also, the names of "causal" vs "original" were confusing. I think this is because the fact that the proposed method augments existing curriculum generation methods was presented only in a single sentence towards the end of the intro, and the knowledge was assumed in the rest of the paper. Seeing as this information is crucial to understand the paper, perhaps the authors can add this crucial piece
2. Figure 3: It's confusing that figure (a) and (b) are identical except for the edit indicators. I needed to visually trace all arrows of both figures to verify this. The presence/absence of edit indicators should be explained in the caption, and the figure can be modified so that it is much quicker for the reader to notice that the only addition is the edit indicators (perhaps through the use of opacity / shading).”

Thanks for your suggestion! We agree that the caption should be self-contained for ease of skimming. We have improved them and added a method flow chart to the main text.

3.1 We agree that the caption in Figure 5 doesn’t stand alone, and we have improved the wording in the revised version. Still, we would like to clarify that we also have provided detailed descriptions of the figure in the experiment section that we compare the performance between “the original curriculum generators” and “the causal version of these generators” using “four best-performing curriculum generators” (four columns). In Figure 5, “causal” refers to the augmented version of the baseline curriculum generators, and “original” refers to the unaugmented, original version of the baseline curriculum generators. The first row is the performance in the Colored Sokoban, and the second row is the performance in the Button Maze. Each column is a different curriculum generator we tested.

3.2 Thanks for pointing this out, we have adjusted the figure.

Weakness 3 & 4:

"Comments on experiments: In general, I believe the environment tested in this paper are rather naive.” ”The authors implement ALP-GMM by fixing the color. I don't think this is a good way to construct baseline. One can run ALP-GMM as a task sampler on a fixed set of tasks. ALP-GMM essentially assigns a probability for different tasks at the each round. If one task has the underlying U that is not aligned with the target task, ALP-GMM will adaptively reduce the weight for that task. I understand that fixing color matches the general story of the whole paper. However, I don't think this is a natural choice in examples where we have access to a simulator. It would be interesting to demonstrate a non-trivial example where C is a proxy of the underlying U and choosing C by running ALP-GMM could be harmful.” “An important way to improve the paper is to propose an algorithm for large (or continuous) state and action space. The current version seems to only work with tabular MDPs, which limits the complexity of environments the algorithm can be applied to.”

We agree that these two environments may seem relatively naive from a human perspective. However, this fact actually strengthens the message of our paper. Namely, if deep reinforcement learning agents cannot solve such seemingly simple problems perfectly (figure 5, trained directly in target task) and if using curriculum learning without incorporating causal knowledge can sometimes be detrimental, how can we expect these agents to achieve similar goals in more challenging environments?

For the ALP-GMM experiments, we would like to clarify that we didn’t force any of the curriculum generators to fix the box color. In the context space, all curriculum generators have the option to “not fix the box color.” Thus, they are already running under your suggested scenario where “C is a proxy of the underlying U, and choosing C by running ALP-GMM (also true for other baseline curriculum generators) could be harmful.” You can also see the following chart summarizing the portion of misaligned source tasks against aligned source tasks during training.

The reason those curriculum generators fail to avoid such harmful source tasks adaptively is that there is still a non-trivial chance (in this setting, $P(U=1)=1/4$) that the agent can get rewarded for pushing an intervened blue color box (undesired behavior). There is also a high probability ($P(U=0)=3/4$) that the agent gets punished for pushing an intervened yellow color box (desired behavior).

Our theorems/algorithms are non-parametric, which means we don’t restrict the parametric form or the size of the scope of those state/action variables neither the distribution over exogenous conditions. They readily apply to high-dimensional (large) or continuous domains. From the experiments, all observations are in the pixel space, which is high-dimensional. Also, note that the environments we tested are not MDP due to the existence of confounders.

Our method is an augmentation to the existing curriculum generators, preparing them for domains where unobserved confounding cannot be ruled out a priori. It does not contradict but rather complements these generators, and is applicable in the same environments as they can support. We agree that showing an experiment in a continuous domain would make the contributions of the paper clearer. Therefore, we have included an extra experiment featuring continuous state and action variables to empirically demonstrate the effectiveness of our method. For the results, please refer to the common response to all the reviewers.

Question 1:

“In Algorithm 3, how is the actions sequence $X_1, \dots, X_H$ obtained in the first place? If we have this optimal action sequence before the learning starts, why do we need RL?”

Action sequence $X_1, \dots, X_H$ follows the topological ordering of these variables in the environment. This representation is common in RL literature. For instance, consider a finite-horizon POMDP with H steps; its actions are ordered as $X_1, X_2, \dots, X_H$, following the time steps that the system dynamics unroll. In other words, in POMDP models, the topological ordering of actions is fixed a priori. RL algorithms are used to find an optimal policy determining the values of these actions.

Question 2:

“In Theorem 3, one of the conditions is that: For every $j=1,\dots,N-1$, actions $\boldsymbol{X}^{(j1)}\subseteq \boldsymbol{X}^{(j+1)}$. Does this mean that the transition must be deterministic and there only exists one unique optimal solution?”

The set of actions $\boldsymbol{X}^{(j)}$ is a set of random variables. Each action in this set is still a random variable, not a specific value. More specifically, condition (i) in Theorem 3 requires that each source task must be causally aligned with a set of actions that is the set of actions whose optimal decision rules are the same as those in the target task. Then, to avoid forgetting (Fig. 4), condition (ii) further requires that the curriculum orders those source tasks in a way that the set of causally aligned actions is expanding along following the curriculum. Combined together, we have the conditions for causally aligned curricula that is implemented as an algorithm in Algo. 3.

In general, Theorem 3 identifies an effective ordering of source tasks so that the learner can converge to the optimal target policy without forgetting useful skills (Fig. 4). However, the transition distribution in each source task is not deterministic but stochastic, and it does not assume a unique optimal solution. Thus, this condition doesn’t indicate that the transition is deterministic or there only exists one unique optimal solution. We hope this clarifies the issue.

Question 1

“Re. the Colored Sokoban example: In the baseline, the context/task space is restricted to tasks where the box color remains constant throughout the game. However, considering a more extensive task space that encompasses all possible combinations of box colors might lead the state-of-the-art curriculum strategies to automatically select relevant/aligned tasks during training. In contrast, the proposed causal curriculum limits the task space to initial agent and box positions, with the environment determining box colors based on intrinsic randomness. Formally, let us denote the initial positions of the agent and the box as $(a_0^x, a_0^y)$ and $(b_0^x, b_0^y)$, respectively. Furthermore, let $c_t$ represent the color of the box at time step $t$, and designate $H$ as the maximum number of steps allowable in the game. In the results presented, the baseline methodology confines the context/task space to instances of the form $[a_0^x, a_0^y, b_0^x, b_0^y, c_0=c_1=\cdots=c_H]$. The extensive task space contains all possible task configurations denoted as $[a_0^x, a_0^y, b_0^x, b_0^y, c_0, c_1, \cdots, c_H]$. The proposed causal curriculum imposes a task space restriction, characterized by tasks of the form $[a_0^x, a_0^y, b_0^x, b_0^y]$, delegating the selection of $[c_0, c_1, \cdots, c_H]$ to the environment, predicated upon intrinsic randomness. Please provide your insights on this aspect.”

For this more extensive task space that encompasses all possible combinations of box colors, those state-of-the-art curriculum generators will still fail to select causally aligned source tasks. Because in this Colored Sokoban, as long as the box color is intervened by the outside curriculum generator, it will be a misaligned source task (as indicated by our theorem), no matter how the color is intervened. Specifically, in example 1, comparing equation 3 against equation 4 (there was a typo here and we have fixed it in the revised version), we see that in target task, the optimal policy is maximizing the reward distribution of, $P\left(Y_i = 10 \big| B_i = \text{next to goal}, C_i, \operatorname{do}\left(X_i = \text{push}\right)\right) = 1[C_i = 0]$, and $P\left(Y_i = -10 \big| B_i = \text{next to goal}, C_i, \operatorname{do}\left(X_i = \text{push}\right)\right) = 1[C_i=1]$.

However, in the color-intervened source task (yellow box), the optimal policy is maximizing the reward distribution of, $P\left(Y_i = 10 \big| B_i = \text{next to goal}, \operatorname{do}\left(C_i = 0, X_i = \text{push}\right)\right) = \frac{1}{4}$, and $P\left(Y_i = -10 \big| B_i = \text{next to goal}, \operatorname{do}\left(C_i = 0, X_i = \text{push}\right)\right) = \frac{3}{4}$. If we intervene the color to be blue, similarly, we have, $P\left(Y_i = 10 \big| B_i = \text{next to goal}, \operatorname{do}\left(C_i = 0, X_i = \text{push}\right)\right) = \frac{1}{4}$, and $P\left(Y_i = -10 \big| B_i = \text{next to goal}, \operatorname{do}\left(C_i = 0, X_i = \text{push}\right)\right) = \frac{3}{4}$. We see that for a curriculum composed of source tasks intervening on the box color, the optimal policy learned is maximizing the reward under the last two distributions instead of the target task distribution. Thus, this more extensive task space also won’t help. Intuitively, because the reward and the box color are confounded by an unobserved random variable (intrinsic randomness), the agent can only access this information by observing the box color. Once the box color is intervened externally, the agent will lose access to this “intrinsic randomness” and cannot know precisely the right time to push the box. In conclusion, even if the baseline methods change the box color at each time step, they still cannot train the agent successfully. From a theoretical perspective, and as characterized by our theory, there are some variables we should not intervene on during the curriculum generation at all; box color in this case falls exactly into this category. We appreciate the opportunity to clarify this sublet and fundamental matter. One of the primary goals of the theoretical framework proposed in this paper is to identify and characterize the variables that cannot be altered without negatively impacting the agent's convergence following training, which constitutes a causally aligned curriculum.

[1] Bareinboim, Elias, et al. "On Pearl’s hierarchy and the foundations of causal inference." Probabilistic and causal inference: the works of Judea Pearl. 2022. 507-556.