Proximal Curriculum with Task Correlations for Deep Reinforcement Learning

Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses to your comments.

1. Extending the presented curriculum analysis for the contextual bandit setting to a general RL setting.

Our investigation into developing a curriculum strategy for the multi-task RL setting (outlined in Section 2) begins by observing that the following three essential factors must be taken into account when selecting a source-target task pair for the student component: (i) Learning potential in the source task, (ii) Transfer potential of the source task, i.e., similarity with the target task, and (iii) Potential performance gain in the target task.

The challenge lies in integrating these quantities to devise an effective curriculum strategy. In pursuit of this goal, we opt to analyze a contextual bandit setting. Intriguingly, our analysis leads us to an intuitive curriculum strategy (Eq. (1)), involving the geometric mean of the three aforementioned factors. We extend this form to general learning settings with natural modifications. Furthermore, our empirical investigation demonstrates the effectiveness of our curriculum strategy.

In the revised version of this paper, we present a high-level approximation for the expected improvement in the training objective in a general RL setting (Section 3.1). Additionally, in the Appendix C.2, we conduct an extra analysis of a tree-structured contextual MDP setting. This analysis underscores the significance of the ZPD-related terms in our proposed curriculum strategy.

While it remains intriguing to delve into more complex learning settings and explore the potential for uncovering richer curriculum strategies, we acknowledge the general absence of such analyses in the existing literature.

2. The metric for task similarity is a bit specific to the tasks. Currently, the metric is defined based on the L2 distance of context parameters.

We thank the reviewer for pointing this out; it is indeed an interesting observation. In our work, we design our curriculum strategy to be agnostic to the task similarity metric, allowing for the application of any such metric. Nevertheless, for the purpose of the empirical evaluation, we opted to employ a standard similarity metric consistently across all domains. Exploring more generalized similarity metrics spanning various domains or even learning the similarity metric itself represents an interesting avenue for future research.

3. Clarification regarding the narrow performance gap between ProxCoRL and ProxCoRL-Un in the PointMass-s:2G environment with a non-uniform target distribution.

Based on the experimental results presented in the paper, we hypothesize that, for the PointMass-s:2G environment, the first two quantities (1) and (2) in Eq. (1), aligned with the ZPD principle, are enough to shift the training task distribution toward harder tasks. This observation aligns with similar trends identified in [1]. Through this shift towards harder tasks, the training task distribution ultimately converges to the target task distribution, even without explicit knowledge of the latter.

Conversely, in the PointMass-s:1T environment, where the target distribution is non-uniform, we observe that the performance gap between ProxCoRL and ProxCoRL-Un increases. Specifically, ProxCoRL converges more rapidly to the target distribution compared to ProxCoRL-Un. This indicates that, in the single target task setting, the incorporation of terms (3) and (4) intensifies the preference for tasks from the target distribution. This increased preference contributes to a more substantial distribution shift towards the target distribution, consequently improving the agent's speed in solving the target task.

[1] Georgios Tzannetos, B´arbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal Curriculum for Reinforcement Learning Agents. In TMLR, 2023.

We hope that our responses can address your concerns and are helpful in improving your rating. If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you! Thank you again for the review.

Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses to your comments.

1. Extending the presented curriculum analysis for the contextual bandit setting to a general RL setting.

Our investigation into developing a curriculum strategy for the multi-task RL setting (outlined in Section 2) begins by observing that the following three essential factors must be taken into account when selecting a source-target task pair for the student component: (i) Learning potential in the source task, (ii) Transfer potential of the source task, i.e., similarity with the target task, and (iii) Potential performance gain in the target task.

The challenge lies in integrating these quantities to devise an effective curriculum strategy. In pursuit of this goal, we opt to analyze a contextual bandit setting. Intriguingly, our analysis leads us to an intuitive curriculum strategy (Eq. (1)), involving the geometric mean of the three aforementioned factors. We extend this form to general learning settings with natural modifications. Furthermore, our empirical investigation demonstrates the effectiveness of our curriculum strategy.

In the revised version of this paper, we present a high-level approximation for the expected improvement in the training objective in a general RL setting (Section 3.1). Additionally, in the Appendix C.2, we conduct an extra analysis of a tree-structured contextual MDP setting. This analysis underscores the significance of the ZPD-related terms in our proposed curriculum strategy.

2. The proposed algorithm is not obviously better than the baselines.

In the empirical evaluation we made, the proposed curriculum is better or at least comparable with the baselines across all the environments.

In the SGR environment, Fig (2a), ProxCoRL outperforms the other techniques.

In the PointMass-s:2G environment, Fig (2b), we can observe a strong performance of ProxCoRL-Un, equivalent to the proposed technique. We hypothesize that, for the PointMass-s:2G environment, the first two quantities (1) and (2) in Eq. (1), aligned with the ZPD principle, are enough to shift the training task distribution toward harder tasks. This observation aligns with similar trends identified in [1]. Through this shift towards harder tasks, the training task distribution ultimately converges to the target task distribution, even without explicit knowledge of the latter.

In the PointMass-s:1T environment, Fig (2c), where the target distribution is non-uniform, we observe that the performance gap between ProxCoRL and ProxCoRL-Un increases. Specifically, ProxCoRL converges more rapidly to the target distribution compared to ProxCoRL-Un. This indicates that, in the single target task setting, the incorporation of terms (3) and (4) intensifies the preference for tasks from the target distribution. This increased preference contributes to a more substantial distribution shift towards the target distribution, consequently improving the agent's speed in solving the target task. CURROT and GRADIENT, both of which are curriculum strategies capable of converging to a single target, demonstrate effectiveness in this environment. Notably, ProxCoRL exhibits a faster convergence rate.

In the BipedalWalker environment, Fig (2d), the uniform nature of the target distribution suggests that ProxCoRL-Un should perform well in this setting. Interestingly, despite the uniform target distribution, ProxCoRL performs comparably to ProxCoRL-Un.

3. The experiments are only conducted on relatively simple tasks with state-based policy.

The experiments are carried out in environments commonly employed in state-of-the-art curriculum techniques ([2], [3]). For example, the BipedalWalker environment features a high-dimensional state space in $R^{24}$ and incorporates LIDAR measurements.

4. Rationale behind replacing $V^\*()$ with $V_{\max}$ in the practical curriculum strategy.

In the general practical setting, we simplify by assuming that the optimal value $V^\*()$ of the environment can be replaced with the maximum achievable value $V_{\max}$ within this environment. In goal-based sparse reward reinforcement learning environments, this value is commonly set to $1$. In domains where obtaining a prior estimate of $V_{\max}$ proves challenging, a practical approach involves dynamically estimating it during training. This is achieved by continuously monitoring and updating the running maximum of the observed returns.

[1] Tzannetos et al. Proximal Curriculum for Reinforcement Learning Agents. In TMLR, 2023.

[2] Klink et al. Curriculum Reinforcement Learning via Constrained Optimal Transport. In ICML, 2022.

[3] Romac et al. Teachmyagent: a Benchmark for Automatic Curriculum Learning in Deep RL. In ICML, 2021.

We hope that our responses can address your concerns and are helpful in improving your rating. Thank you again for the review.

Thank authors for the detailed response. I appreciate your efforts on the newly updated theoretical results in C.2. However, I'm still concerned that the proposed method is not significantly better than baselines and the theoretical contribution is super limited to over-simplified cases. Thus, I'd keep my score.

Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses to your comments.

1. Use “citep” instead of “citet” for citations where the citation is not part of the sentence… Minor: Figure 2 plotlines are a bit thick, making it somewhat difficult to read. I would suggest slightly decreasing the line width.

We thank the reviewer for these comments. We have updated the PDF with the suggested changes.

If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you! Thank you again for the review.

Thank you for carefully reviewing our paper! We greatly appreciate your feedback. Please see below our responses to your comments.

1. Difference between the proposed method and the one presented in prior work [1].

The setting considered in the prior work [1] differs notably. Specifically, it does not account for a target task distribution, opting instead to gauge the agent's performance relative to a uniform distribution of tasks across the context space. In our work, the curriculum is developed with consideration for a target task setting, resulting in the introduction of additional terms (3) and (4) in Eq. (1). These terms enhance the preference for selecting tasks correlated with the target task.

2. Extending the presented curriculum analysis for the contextual bandit setting to a general RL setting.

Our investigation into developing a curriculum strategy for the multi-task RL setting (outlined in Section 2) begins by observing that the following three essential factors must be taken into account when selecting a source-target task pair for the student component: (i) Learning potential in the source task, (ii) Transfer potential of the source task, i.e., similarity with the target task, and (iii) Potential performance gain in the target task.

The challenge lies in integrating these quantities to devise an effective curriculum strategy. In pursuit of this goal, we opt to analyze a contextual bandit setting. Intriguingly, our analysis leads us to an intuitive curriculum strategy (Eq. (1)), involving the geometric mean of the three aforementioned factors. We extend this form to general learning settings with natural modifications. Furthermore, our empirical investigation demonstrates the effectiveness of our curriculum strategy.

In the revised version of this paper, we present a high-level approximation for the expected improvement in the training objective in a general RL setting (Section 3.1). Additionally, in the Appendix C.2, we conduct an extra analysis of a tree-structured contextual MDP setting. This analysis underscores the significance of the ZPD-related terms in our proposed curriculum strategy.

While it remains intriguing to delve into more complex learning settings and explore the potential for uncovering richer curriculum strategies, we acknowledge the general absence of such analyses in the existing literature.

3. The proposed method requires domain-specific hyperparameter tuning of $V_{\max}$.

$V_{\max}$ denotes the maximum attainable value in the given environment. In goal-based sparse reward reinforcement learning environments, this value is commonly set to $1$. In domains where obtaining a prior estimate of $V_{\max}$ proves challenging, a practical approach involves dynamically estimating it during training. This is achieved by continuously monitoring and updating the running maximum of the observed returns.

4. Clarification regarding the narrow performance gap between ProxCoRL and ProxCoRL-Un in the case of a non-uniform task distribution (PointMass-s:2G).

Based on the experimental results presented in the paper, we hypothesize that, for the PointMass-s:2G environment, the first two quantities (1) and (2) in Eq. (1), aligned with the ZPD principle, are enough to shift the training task distribution toward harder tasks. This observation aligns with similar trends identified in [1]. Through this shift towards harder tasks, the training task distribution ultimately converges to the target task distribution, even without explicit knowledge of the latter.

Conversely, in the PointMass-s:1T environment, where the target distribution is non-uniform, we observe that the performance gap between ProxCoRL and ProxCoRL-Un increases. Specifically, ProxCoRL converges more rapidly to the target distribution compared to ProxCoRL-Un. This indicates that, in the single target task setting, the incorporation of terms (3) and (4) intensifies the preference for tasks from the target distribution. This increased preference contributes to a more substantial distribution shift towards the target distribution, consequently improving the agent's speed in solving the target task.

[1] Georgios Tzannetos, B´arbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal Curriculum for Reinforcement Learning Agents. In TMLR, 2023.

We hope that our responses can address your concerns and are helpful in improving your rating. If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you! Thank you again for the review.