Efficient Multi-task Reinforcement Learning with Cross-Task Policy Guidance

Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.

Scalability to complex environments.

We have tested CTPG on MetaWorld and HalfCheetah, the commonly used and authoritative benchmarks in the MTRL community, covering two main fields of manipulation and locomotion. In both two environments with diverse task settings, CTPG has shown outstanding performance, which demonstrated that CTPG can effectively learn policy sharing between tasks, thus improving the sample efficiency of MTRL. In addition, CTPG is a general MTRL framework that can be adapted to other MTRL environments and other RL algorithms. We conducted experiments using another base RL algorithm TD3 in Fig.3 of the global Rebuttal Comment PDF, and the results demonstrate that CTPG is equally applicable to other algorithms to enhance sample efficiency. CTPG can also be migrated to other MTRL benchmarks without additional configuration and deployment. If you have a recommendation for another benchmark, we'd be happy to try it on there too :)

Computational requirements and computational complexity.

We provided detailed information about the computational resources used in our experiments in Appendix E.1. For computational complexity, we show the training curves with the x-axis being wall-clock time on the MetaWorld-MT10 environment in Fig.4 of the global Rebuttal Comment PDF. The results indicate that although CTPG takes longer to train, it still outperforms baselines for the same training time. Furthermore, during the evaluation and deployment phase, CTPG's guide policy will no longer be used, so no additional execution time and storage space is required.

Discussion on potential negative societal impacts.

We provided Broader Impacts in Appendix F. CTPG is a general MTRL framework and does not introduce additional social impacts that need to be discussed. If you feel there are any necessary social implicates that need to be discussed, welcome to point them out, and we will add discussions about them in the revised manuscript.

Thank you once again for your valuable feedback. We hope our response has satisfactorily addressed your concerns. If you find that we have addressed your concerns, we kindly hope you reconsider your rating. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.

Thank you very much for your constructive feedback and acknowledgment of our efforts.

Floating pieces

1. Hindsight off-policy correction: On the one hand, it is deeply integral to our CTPG framework, instead of just a floating piece. It addresses the unavoidable non-stationarity issue in off-policy training. We demonstrate this in Fig 5c and 11c, where it is helpful and its effect is more pronounced with more tasks, reinforcing its necessity within CTPG. On the other hand, it seems that there is a misunderstanding on the reason for proposing this module. The issue you mentioned does indeed exist and is still a matter of interest for the community. In such case, the data is invalid for training because none of the policies can regenerate this old sequence, so the hindsight off-policy correction won't work either. However, our module is not proposed to solve this issue and has its own motivation. It should not be considered a floating piece merely because it does not address a problem outside its intended scope.

2. Action space masking in the policy-filter gate: The integration of this module with CTPG is also essential, which avoids negative policy guidance from unhelpful control policies. Although the guide policy can also be learned by gradient updates only, the policy-filter gate serves to expedite this process by narrowing down the candidate policies, especially when the number of tasks is particularly large, thus further reducing the exploration space and improving sample efficiency. As shown in Fig 5a and 11a, training efficiency will be significantly reduced without this gate. Here we want to emphasize that although the functionality of this gate can be achieved by policy gradient update, our module can significantly improve training efficiency.

3. Temperature coefficient of SAC in the guide-block gate: We aim to use it to filter tasks that have been mastered or converged on. The choice of SAC and its adaptive temperature mechanism was made due to its robust performance in various RL tasks. We want to emphasize that CTPG is a general MTRL framework that can be adapted with other RL algorithms. We explore the adaptation of TD3 in Fig.3 of the global Comment PDF. Using the temperature of SAC as an indicator is indeed an empirical choice. We admit it may not always be true, thus we provide several alternatives not specific to SAC. For example, we try an intuitive metric success rate in Fig 5b and 11b, which shows fair performance with SAC temperature. Besides, policy entropy (in algorithms like PPO) or cumulative rewards (when there is no significant change) can also perform as indicators. The guide-block gate serves as an essential module with various implementations, and we provide several alternatives and welcome further study based on ours.

The three modules collectively contribute to share effective policies between tasks, thus improving the sample efficiency, which is the key goal of MTRL. In addition, these three modules do not involve any hyperparameter settings, so there is no additional burden for tuning. Hope our response can change your initial perception that they were just floating pieces.

Forward time of hindsight off-policy correction and performance with wall-clock time

Compared to the gradient update process, hindsight off-policy correction requires only forward computation, making the time consumed negligible. In addition, the calculation of each control policy's probability for generating the historical action sequence can be parallelized. This is achieved by extending the input of the actor from [batch_size] to [batch_size, task_num], where the second dimension distinguishes the different task representations.

Fig.4 of the global Comment PDF shows the training curves with wall-clock time on MetaWorld-MT10. Although CTPG takes longer to train, it still outperforms baselines for the same training time.

Scenarios that CTPG works

CTPG does not require that a task have to be a sub-strategy of another task. If tasks can naturally form a curriculum as you mentioned, CTPG definitely can learn an effective policy. Even if other tasks' control policies do not perform as well on the current task, CTPG's guide policy can still facilitate an effective mixed policy. In specific, the agent can choose different control policies at different stages within a single episode, as shown in Fig 4 in the paper. In other words, the policy guidance uses a learnable mixture policy consisting of all control policies. Furthermore, referring to Reviewer xTD7's suggestion, we set up a new baseline called BPT (Best Performance Transfer), which selects the best-performing policy in the current task among all task policies to guide the exploration. The results shown in Fig.1 of the global Comment PDF demonstrate that CTPG also outperforms BPT. This is because CTPG's guide policy learns a more flexible strategy, rather than simply relying on a single best-performing policy for guidance.

Here we extend the earlier example involving bicycle and motorcycle for better understanding. Suppose an agent is learning four tasks: unicycle(U), bicycle(B), motorcycle(M), and automobile(A). They do not form a curriculum and need to be learned simultaneously. (B) and (M) share the skill balancing on two wheels. (U) and (B) share the skill pedal-driven, while (M) and (A) share the skill ignition to move. With CTPG, if the agent has learned the pedaling skill in (B), it can transfer this skill to (U), even if it hasn’t yet mastered balancing. Similarly, if the agent has learned balance in (B) and ignition in (A), it can quickly master (M) by combining these two skills.

Minor errors

Thank you for pointing out the typo "dose", we will correct it in the revised manuscript.

Thank you once again for your valuable feedback. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.

Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.

Sample efficiency comparison.

Please refer to Figure 10 (Appendix D.1), which presents the full training curves for our main experiment (Table 1). It shows that beyond the ultimate performance improvement, CTPG also enhances the sample efficiency.

Comparison to K-step QMP.

In our experiments, we use the hyperparameter settings from the original QMP paper. Thus, the QMP in our experiment is exactly the K-step QMP ($K$ = 10 in implementation) you asked for, which as you mentioned, shows that the performance gain comes from other designs of the proposed method. We are sorry for the ambiguity and we will add this hyperparameter setting to the Appendix in the revised manuscript. Thank you again for pointing this out.

Hyperparameter guide step $K$.

Guide step $K$ is a necessary hyperparameter introduced by CTPG and may need slight adjustment across testbeds. However, we want to emphasize that the improvement gains of CTPG do not rely on the careful tuning of $K$ at all. As shown in Fig.2 of the global Rebuttal Comment PDF (a clear version of Figure 12), the performance improvement is evident even without tuning $K$. If the user pursues the potential best performance, we recommend exploring the value of $K$, and it can be selected easily with the help of hyperparameter search methods, such as Bayesian Optimization, Hyperband, etc. In the Conclusion and Discussion Section, we have already indicated that our future work will focus on dynamically selecting the guide step $K$.

In addition, the output of CTPG's guide policy can add another action dimension to choose a specific guide step $K$. This approach not only automatically selects the $K$ value in different environments but also dynamically selects it at different time steps. We already have preliminary experimental results demonstrating the effectiveness of this scheme, and it also shows the high scalability of the CTPG framework. However, we believe that the contribution could be a new work, so it is mentioned only in future work.

Adaptability of other RL algorithms to CTPG.

CTPG is a general MTRL framework that can be adapted with other RL algorithms, even allowing the control and guide policy to use different RL algorithms. Since SAC is a widely used algorithm in continuous control and serves as the base algorithm for all baselines, we also choose SAC as the base algorithm in this work. In addition, we explore the adaptation of TD3 with the CTPG and also employ different RL algorithms for the control and guide policies. Specifically, the control policy uses TD3, and the guide policy uses DQN. This approach is compared with the TD3-based MTRL method, and the result is displayed in Fig.3 of the global Rebuttal Comment PDF. The result shows that CTPG can enhance performance and sample efficiency when combined with other backbone RL algorithms.

Section 4 is difficult to follow.

Thank you very much for your suggestion. We will include an additional intuition before introducing each module to enhance readability in the revised manuscript.

Similarity with QMP.

We highly respect QMP and believe that QMP is an excellent work that greatly inspired our research. We discussed their differences in Related Work Section. Moreover, QMP serves as the most important and the only baseline in our experiments, and we openly compared our method against it. Following your suggestion, we will include a more detailed introduction of QMP and its great contribution in the revised Introduction.

Here, we would like to re-emphasize the difference between CTPG and QMP: QMP uses the maximum Q-value of a single step to select the shared behavior over continuous K steps, which only guarantees the optimality of the first step of the shared policy. In contrast, CTPG learns a guide policy with the action space being exactly the task control policies to identify useful sharing policies for guidance by considering the benefits over K steps collectively. Additionally, CTPG proposes two gating mechanisms to avoid negative transfer from unhelpful policies, thereby improving sample efficiency.

Thank you once again for your valuable feedback. We hope our response has satisfactorily addressed your concerns. If you find that we have addressed your concerns, we kindly hope you reconsider your rating. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.

Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.

Additional comparison with a simple transfer learning strategy where experience is only transferred from the policy that has the best performance.

[1] focus on continual RL, requiring a predefined task learning sequence, which does not match the experimental setting of MTRL, so we cannot directly compare it with CTPG and other baselines. Therefore, we implement a comparable version based on the core idea of [1] and your description: selecting the policy that performs best in the current task among all task policies to guide exploration in generating trajectory data. Specifically, after each $H$ rounds of data collection, we evaluate all policies across all tasks and choose the best-performing policy for each task to collect the data for the next $H$ rounds. We refer to this baseline as BPT (Best Performance Transfer) and set $H$ to 50 episodes in implementation. We use MTSAC in HalfCheetah-MT8 and MHSAC in MetaWorld-MT10. The result, shown in Fig.1 of the global Rebuttal Comment PDF, demonstrates that CTPG outperforms BPT. CTPG's guide policy learns a more flexible strategy: it not only can learn to share a single policy within a complete trajectory (like BPT), but also can develop a mixture policy by combining different control policies, making it more transferable between tasks. Notably, BPT performs better in HalfCheetah-MT8 than in MetaWorld-MT10 because the tasks in HalfCheetah-MT8 share more significant similarities, whereas the policy transfer and guidance in MetaWorld-MT10 require combination strategies.

[1] Disentangling Transfer in Continual Reinforcement Learning. Wolczyk et al. NeurIPS 2022.

The effect of the hindsight off-policy correction mechanism on SAC's actor network training.

The hindsight off-policy correction mechanism affects not only the critic update in SAC but also the actor update. Although the actor loss (Line 4 in Algorithm 2) does not explicitly use relabeled $j'_t$, it indirectly affects actor learning due to SAC's unique actor update method [2]. Specifically, SAC's actor loss is derived from its optimization objective, and the actor policy is updated according to:

$\pi_{new} = \arg\min_{\pi' \in \Pi} D_{KL} \left( \pi'(\cdot | s_t) \left\| \frac{\exp\left(\frac{1}{\alpha} Q^{\pi_{old}}(s_t, \cdot) \right)}{Z^{\pi_{old}}(s_t)} \right. \right),$

where the partition function $Z^{\pi_{old}}(s_t)$ normalizes the distribution. In essence, SAC's policy is updated by fitting the distribution of the SoftMax function of Q-value with temperature $\alpha$. Therefore, modifying the update of $Q(j_t|s)$ to $Q(j'_t|s)$ using hindsight off-policy correction leads to a different actor optimization objective, thus affecting the training of the actor network. We will also elaborate on how the hindsight off-policy correction mechanism affects guide policy training in the revised manuscript.

[2] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Haarnoja, Tuomas, et al. ICML 2018.

Thank you once again for your valuable feedback. We hope our response has satisfactorily addressed your concerns. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.

Thank you for considering attempting to address the points I've previously raised.

Specifically, after each $H$ round of data collection, we evaluate all policies across all tasks and choose the best-performing policy to collect the data for the next $H$ rounds. We refer to this baseline as BPT (Best Performance Transfer)

This is exactly the baseline that I envisioned in my previous feedback. Given the positive results when comparing against this baseline, I no longer view the lack of comparisons against BPT as a minor weakness of this paper.

Although the actor loss (Line 4 in Algorithm 2) does not explicitly use relabeled $j'_t$, it indirectly affects actor learning due to SAC's unique actor update method [2].

I also agree that this needs to be better highlighted in the next manuscript version. Perhaps it's also important to note that unlike SAC-based actor updates in environments with continuous action spaces (where we rely on a Monte Carlo estimate of the KLD objective since integrating over each possible action is impossible), the KLD objective can be easily evaluated when the input distributions to the KL divergence are categorical.

Closing remarks

Given the authors' responses to the points raised by each reviewer, I am increasing my score because I believe this is a good paper worthy of acceptance.