Efficient Multi-task Reinforcement Learning via Selective Behavior Sharing

We thank the reviewer for the constructive feedback especially regarding clarifications in the writing and additional related works to add. We have incorporated these into the paper. Additionally, we would like to emphasize that the goal of this paper is to advocate for selective behavior sharing as a promising avenue in multi-task RL which can be complementary to (and not in competition with) many of the existing parameter sharing and data sharing methods that are more well known. We address your remaining concerns and questions below.

Weakness C,D: Distinct policy for each task:

We clarify that our algorithm’s design is not limited to distinct policies. In fact, the set of experiments in Section 6.2 show that QMP can be implemented with a single multi-head multi-task policy and that the behavior sharing advantages are complementary with parameter sharing advantages.

Furthermore, the choice of distinct v/s shared task policies depends on the task family. Evidently, parameter-sharing is not always optimal and could even hurt training over distinct policies (See Multistage Reacher results in Figure 5 and [3]). Since we consider task sets where different tasks may require conflicting behavior, we base our primary comparisons on distinct policy networks, but also provide results on parameter-sharing + QMP.

Weakness E: Weak Baselines

We believe the reviewer is referring to the comparisons we make in Section 6.1, which are comparisons with other behavior sharing methods and not multi-task RL methods in general. This supports our argument that QMP is an effective way to share behaviors in multi-task RL problems. Importantly, we are not claiming that QMP, or even behavior sharing, is the best MTRL method on its own, and therefore do not compare with complementary non-behavior sharing methods. In fact, we show QMP can be combined with methods like parameter sharing (Section 6.2) to gain the advantages of both methods.

Weakness F: Lack of notable improvements

We would like to note that reviewers QRge, Zv3o, and neqy all noted that the empirical results were comprehensive and demonstrated the effectiveness of QMP. In our experiments, we chose a wide range of environments and tasks with shared and conflicting behaviors to demonstrate that our simple method of QMP improves sample efficiency and convergence across general multi-task sets, and did not design tasks specifically to favor our method which may have shown more dramatic differences. This is in line with the theoretical proofs in [Appendix G] showing that QMP guarantees a weak monotonic improvement over the baseline of no behavior sharing.

Question C: Comparisons to data-sharing methods

• [Fig 9 (c)] Yu et al. (2021) [1] requires a ground truth reward function to re-label shared data between methods so it is not applicable to our problem setting. In Fig 9 (c), we show that QMP even outperforms CDS which makes an extra assumption of ground truth reward-relabeling.
• [Fig 4] We do compare against CDS’s follow-up work UDS [2], in Section 6.1, which does not require the true reward labels. QMP consistently outperforms UDS.

Question E: Should QMP always choose its own task policy?

[Theorem G.2] While the task policy $\pi_i$ is trained to maximize $Q_i$ over continuous action spaces, it always lags behind $Q_i$, which is itself constantly being updated with new rollouts over training. So the Q-filter can select a different policy if $pi_i$ is not fully optimized for the current state, and a different task policy suggests a better action. When this happens, Theorem G.2 guarantees performance improvement.

Question F: “Fully-Shared-Behaviors” baseline

We use this baseline in section 6.1 specifically to answer “How does QMP compare to other forms of behavior sharing?”, by comparing against a policy that has the same behavior everywhere for all tasks, and therefore mask out the task identifiers to enable policies to learn shared behaviors across tasks. This is intended as a behavior sharing comparison, not an analysis on parameter sharing in single policies which we do compare against in Section 6.2.

[References]

[1] Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Levine, S., and Finn, C. Conservative data sharing for multi-task offline reinforcement learning. NeurIPS 2021.
[2] Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Finn, C., and Levine, S. How to leverage unlabeled data in offline reinforcement learning. ICML 2022.
[3] Yu, Tianhe, et al. Gradient surgery for multi-task learning. NeurIPS 2020.\

We thank the reviewer for the helpful feedback on where we can provide additional intuition for our method and on the structure of the related works. We address your concerns and questions below.

Theoretical guarantees

[Appendix G] Please refer to the combined response, where we add convergence and policy improvement proofs for the QMP Algorithm’s modification due to its mixture policy.

Using Q-function to evaluate shareable behaviors

[Appendix H] Please refer to the combined response, where we justify (based on prior work, mathematical intuition, and empirical results) why temporally-extended behaviors can effectively obtain high-rewarding trajectories, even when the Q-switch is used to evaluate the mixture policy on the current action.

We note that, for a single action proposal from another task policy, the Q-function can indeed evaluate $Q_i(s, \pi_j(s))$ because $\pi_j(s)$ is just another action to evaluate for $Q_i$. We prove the utility of other tasks in Task $i$’s policy improvement step in Theorem G.2 for H=1, i.e., behaviors of length 1. Further justification for rolling out $\pi_j$ for a sequence of future states is provided in Appendix H.

Related Work Discussion

[Section 2] We added several relevant works on MTRL through shared representations and skill learning as a single task behavior sharing example.

We hope the added theoretical results and justifications addresses all your concerns.

We thank you for your thorough review and your positive comments on the direction of our paper and the extensive empirical results and analysis. We address your questions and concerns below.

Theoretical Analysis on the convergence (rate) of QMP

[Appendix G] Please refer to the combined response, where we add convergence and policy improvement proofs for the QMP Algorithm’s modification due to its mixture policy. We prove that QMP’s mixture policy induces a Bellman operator that is a contraction, and QMP further guarantees a weak monotonic policy improvement over its base algorithm, SAC.

Intuition and analysis on why QMP works for behaviors

Q1: Using Q-function to evaluate H-step behaviors

[Appendix H] Please refer to the combined response, where we justify (based on prior work, mathematical intuition, and empirical results) why temporally-extended behaviors can effectively obtain high-rewarding trajectories, even when the Q-switch is used to evaluate the mixture policy on the current action.

Q2: Undesired Behavior Sharing?

We address this in detail in the combined response, but in summary, it is possible for QMP to cause undesired behavior sharing through the Q-filter but is able to recover quickly and self-correct due to our method design which only uses other task policies to gather training data. We see this in effect in Task 4 of Multistage Reacher which completely conflicts with the other tasks but QMP still performs as well as other baselines that do not share behaviors.

Q3: Concerns about bias in learned Q-function

The reviewer correctly points out that the learned Q-function may be biased, leading to QMP selecting sub-optimal policies. However, this is true for any type of Q-learning where a policy tries to pick actions to maximize a learned Q-function (methods based on SAC [1] and Deterministic Policy Gradient [2]). This generally does not hinder performance in any RL algorithm due to self-correction from online data collection. In fact, many performant Q-learning-based RL algorithms, learn policies directly to maximize a learned Q-function [2, 6, 7]. As explained in Section 4.2 in the paper, the learned Q-function is a critical component of many online RL algorithms [1,2] and has even shown to be a useful filter for high-quality training data [3,4,5]. Our results in conflicting tasks (see QMP vs. No-shared in Figure 8, Task 4) demonstrate that QMP does not suffer from Q-function bias any more than the RL algorithm it is built on top of.

Q4: Could QMP slow down the training progress?

As addressed in Q3, QMP is hindered by the same Q-function bias as most Q-learning algorithms but does not impact sample efficiency in any of the task sets we tried even when there are directly conflicting behaviors (see example in Q2).

We hope the combination of a simple method with strong empirical results and analysis, and the added theoretical results and justifications addresses all your concerns.

[References]

[1] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML 2018.
[2] Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv 2015.
[3] Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Levine, S., and Finn, C. Conservative data sharing for multi-task offline reinforcement learning. NeurIPS 2021.
[4] Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Overcoming exploration in reinforcement learning with demonstrations. ICRA 2018.
[5] Sasaki, F. and Yamashina, R. Behavioral cloning from noisy demonstrations. ICLR 2020.
[6] Silver, David, et al. "Deterministic policy gradient algorithms." ICML 2014.
[7] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." ICML 2018.

We thank you for your helpful feedback and pointing us to relevant works to include, that we have incorporated a discussion in the revision. We address your concern about the H-step rollouts in the combined response and the remaining questions below.

1. Shared state-action dimension

This is a common assumption in many multi-task learning works [1-7]. Experiments on MT10 show that the state-action space does not need to be necessarily shared and only the dimensions are shared. Even specialized architectures like Perceiver that can take multimodal and diverse inputs are complementary to our work, if the Q-function is modeled in that way. So, the requirement of same state-action dimensionality is a consequence of the choice of Q-network, not a limitation of behavior-sharing.

2. Out-of-distribution action proposal

Out-of-distribution means that the Q-switch, i.e., $Q_i$ may not yet be well-trained on (s, $a_j$), where the $a_j$ is another task’s proposal. In this case, the Q(s, $a_j$) value might be inaccurate. Even if the Q-switch erroneously selects this action $a_j$ to collect data, it would self-correct because by collecting data with $a_j$, the Q-switch would be more accurate on (s, $a_j$) now.

3a. Theoretical grounds for criterion used to collect data

[Appendix G] Please refer to the combined response, where we add convergence and policy improvement proofs for the QMP Algorithm’s modification due to its mixture policy.

3b. Using Q-function to do temporally extended data collection

[Appendix H] Please refer to the combined response, where we justify (based on prior work, mathematical intuition, and empirical results) why temporally-extended behaviors can effectively obtain high-rewarding trajectories, even when the Q-switch is used to evaluate the mixture policy on the current action.

4. Data Sampling Details

To train the policy and critic for task $i$, we sample uniformly from the data collected in the replay buffer by QMP’s mixture policy $\pi_i^{\text{mix}}$ without needing to do any specialized data balancing or priority sampling based on rewards. This means that the data collected by QMP is helpful for training $\pi_i$ without additional implementation tricks.

Baseline Parameter Sharing Method

In section 6.2, we use a widely used baseline parameter sharing method, instead of more sophisticated methods like the reviewer points out. This is because this experiment was not meant to compare parameter sharing v/s behavior sharing, but to show that they are complementary — which we will clarify in the text. Our goal is not to compare parameters-only v/s behaviors-only, because they are orthogonal and complementary ways to share knowledge in MTRL. Our goal is to show that in simultaneous MTRL, behavior-sharing can augment other ways of sharing.

• Particularly, in Maze Navigation, we see that Parameters-Only already improves significantly over Neither, which shows that methods like PCGrad are not needed in this environment. Our goal is to show that even in this case, Parameters + Behavior sharing further improves significantly over Parameters-Only.
• As the reviewer points out, in Multistage Reacher, because of the potential conflicts in tasks, parameters-only performs poorly than Neither. However, still Parameters + Behaviors improves over parameters-only. In this case, while combinations of QMP and more sophisticated parameter sharing may yield even better results. However, we believe that the multi-head policy we used is sufficient evidence to demonstrate that QMP is complementary with parameter sharing.

We hope this clarifies and addresses the concerns raised.

[References]

[1] Teh et al. Distral. Robust multitask reinforcement learning. NeurIPS, 2017.
[2] Ghosh et al. Divide-and-conquer reinforcement learning. ICLR, 2018.
[3] Yu et al. Conservative data sharing for multi-task offline reinforcement learning. NeurIPS, 2021.
[4] Yu et al. How to leverage unlabeled data in offline reinforcement learning. ICML, 2022.
[5] Kalashnikov et al., Scaling up multi-task robotic reinforcement learning. CoRL 2021b.
[6] Reed, Scott, et al. "A Generalist Agent." TMLR 2022.
[7] Jaegle, Andrew, et al. Perceiver IO: A General Architecture for Structured Inputs & Outputs. ICLR 2021.