Knowledge Transfer through Value Function for Compositional Tasks

Cons：

• Placing the statement 'Our main contribution is demonstrating the effectiveness...' in the abstract section creates confusion regarding the primary contribution of the article, which could be expressed more explicitly.
• The comparison between this article and existing work can be analyzed in detail in the introduction section, which helps readers understand the contribution of this article.
• The paper lacks sufficient information on implementation details, such as the generation of source tasks and hyperparameter settings.
• Too few ablation experiments and comparative experiments have made me doubt the scalability of the method, such as the difficulty of adjusting $\alpha$ and $\beta$ between different tasks.

1. It would be very helpful to improve the writing to make it clear what is the technical contribution.
2. It is hard to tell what are the formal and quantitative definition of the task complexities. The paper seems particularly ambitious to cover as many variations as possible while somehow looses the focus. The validation seems solely focusing on some of the task variations while not demonstrating on transferring between different state/action spaces.
3. The validation assumes the existence of a strategy to generate a curriculum, which is actually central to curriculum learning. If the investigation is about what transfer strategy to use, it would make more sense to benchmark other competing transfer strategies. The baselines like \pi_1 look weak.
4. The results are hard to interpret their significance. That is probably due to visualisation, e.g. in Fig. 4, showing extremely large variance. It is hard to tell the competing methods apart due to the overlap. It might be more readable to use table to report statistics at a selected set of episode steps.
5. The benchmarking tasks could be stronger. Curriculum learning has demonstrated in learning with more complex environment dynamics. Demonstrating efficacy could significantly improve the paper's soundness and significance.

• The motivation and the contributions of this work are not clear to me. While curriculum learning is certainly an important topic within reinforcement learning, I do not understand the approach being proposed in this paper. On top of the many technical concerns I have (see below), I don't understand why such a knowledge transfer mechanism is needed. If the state-action space is the same between tasks, why not simply train each task until convergence (or some other training heuristic) and then move on to training the subsequent task in the curriculum as is commonly done? If the state-action space is not the same, then this method will also need to leverage some method to deal with it just as standard curriculum learning would (e.g. through bootstrapping, action advising, policy distillation, etc).
• The technical details of the proposed approach are also not clear to me. Early in training in the target task, the target Q-function will not have converged and will have poor estimates. And if we use the Q-function from the source task, wouldn't it be out-of-distribution in the target task? In addition, why would we assume that the Q-value in the source task is a proxy for a good action in the target task, when the tasks differ? Under what conditions is this expected to work? Why do this sort of blending at all, why not just start with the policy from the source task and perform standard RL training in the target task?
• The experimental results do not include comparisons to any sort of baseline, including a standard curriculum training setup where policies are simply trained to convergence in a sequence of tasks. The figures are also not clear to me (e.g. Figure 3), I am not sure what I am supposed to take away from them and they are quite difficult to read. For example, the text says the learning process is significantly enhanced but what indicates this?

The main contribution of the paper is to adjust the degrees of freedom for knowledge transfer yet the empirical results are focused on the proof-of-concept rather than varying degree of freedom, and how that impacts the algorithm performance. There are no theoretical proofs to strengthen the paper in this regard. It feels like the paper is incomplete without supporting evidence whether empirical or theoretical.

However, the reviewer finds this paper hard to follow, and it requires significant improvements in clarity.

1. A lot of technical details are missing or need to be clarified. For example, a) the paper mentioned that 'source tasks are sampled based on their complexity' many times. But what's the sampled policy? b) Also, the paper mentioned that 'the state space is different, then a function $s_{\mathcal{M}}(\phi) = h(s_{\mathcal{M}_t}(\phi))$ is needed'. What does $h(\cdot)$ define? How to learn $h(\cdot)$? Is it pre-defined by humans? What's $\phi$? c) In Section 4.2, does the equation update the source Q-functions? Why? d) The formula '$\alpha +\beta=1$ and $\alpha>\beta=1$ ' does not make sense. e) The way to combine a set of Q-functions into one source Q-function is not clear.

2. The contents are inconsistent. For example, 'self-correction' and 'self-correcting' are mixed-used.

3. Some typos exist in the paper, making it hard to understand.

4. The figures are hard to read.

No limitations are discussed in this paper. For example,

1. Using $\max(Q)$ makes the paper not able to extend to continuous action spaces.
2. The curriculum is defined by humans, which is infeasible to be applied to complex domains, requiring a certain amount of expert domain knowledge.
3. There are a lot of related works doing the same thing as this paper [1-3], while they automatically generate the curriculum.

[1] Emergent complexity and zero-shot transfer via unsupervised environment design - Dennis et al.NEURIPS 2020

[2] Evolving curricula with regret-based environment design - Parker-Holder et al. ICML 2022

[3] Understanding the complexity gains of single-task rl with a curriculum. ICML 2023.