QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing

We thank the reviewer for their detailed feedback and constructive comments. We are pleased that you found the paper well-written and the results promising. Below, we address your specific concerns point by point. We also made revisions to clarify the motivation and contributions of QMP based on your suggestions.

it's not intuitively clear why QMP works as well as it does. I have some concerns about the approach of naively taking the maximum over per-task Q-functions—this may not be suitable in general settings, such as more stochastic environments, as it could lead to issues like switching tasks at every step or favoring certain tasks while neglecting others, which is undesirable.

We appreciate this observation and use Section 5 to clarify the mechanism why QMP works.

Why QMP works:

QMP works because at certain states, a task’s policy may lag behind its corresponding Q-function, failing to fully optimize the SAC objective:
\ $

\pi^{\text{new}} = \arg \min_{\pi' \in \Pi} D_{\text{KL}} \left( \pi'(\cdot \mid s_t) \,\middle\|\, \frac{\exp\left(\frac{1}{\alpha} Q^{\pi^{\text{old}}}(s_t, \cdot)\right)}{Z^{\pi^{\text{old}}}(s_t)} \right)
\ $

So, QMP leverages other task policies $\pi_j$ that may have already learned optimal or near-optimal behaviors for similar states. By selecting the policy $\pi_j$ that maximizes the above objective at a given state, QMP improves over the SAC objective, leading to greater policy improvement (Theorem 5.1) and the sample efficiency gains observed empirically.

Why QMP is Stable:

We would like to clarify that QMP does not “switch tasks at every step”. In one episode, the task stays fixed and the data is collected with its associated $\\pi\_i^{mix}$. At a given state, $\\pi\_i^{mix}$ selects the particular action-distribution p(a) \= \\pi\_j(a|s) that collects data. Finally, the gathered data is all used to train $\\pi\_i$ only. Each task and policy receives equal episodes of data collection and gradient updates. This design avoids favoring certain tasks while maintaining stability (see Algorithm 1 for a formal description).

Why naive maximization in Eq. (3) is sound, even in stochastic environments

We understand it may be counter-intuitive, but the naive maximization step follows exactly from the policy improvement theorem, and is exactly what SAC optimizes. Appendix C derives the maximization step directly from the definition of soft policy improvement in SAC.

The maximization step is robust in stochastic environments, because it is executed at every state and the Q-function accounts for future stochasticity. Concretely, at state $s\_k$, the selection is governed by $\\mathbb{E}\_\pi [Q(s\_k, a)]$. Under stochastic dynamics, the next state is $s\_{k+1} \\sim \\mathcal{T}(s\_k, a)$. Here, the selection will be governed by $\\mathbb{E}\_\pi [Q(s\_{k+1}, a)]$. So, since QMP is evaluated at every state, it ensures that QMP's mechanism is robust to stochastic environments.

We thank the reviewer for their insightful feedback and constructive criticisms, and we address your concerns point by point below.

As an overarching comment, our stated primary contributions are: introducing behavior policy sharing as a new avenue of information sharing in MTRL, proposing QMP as a simple and principled method for this purpose, and demonstrating its complementary gains to various MTRL settings. We do not claim QMP is a standalone state-of-the-art MTRL method but rather a plug-and-play component that can be integrated with existing MTRL frameworks, including parameter sharing methods like Sodhani et al. $1$ . To this end, we hope our experiments sufficiently support our claims, showing complementary gains to existing MTRL and comparisons with baseline behavior-sharing methods.

1. Clarify scope and compare information sharing strategies

1. This paper only proposes a multi-task data sampling strategy implemented within an off-policy framework, which is limited in scope and shows limited improvement in more complex tasks, such as MT10 and MT50.

We agree that QMP is a multi-task data sampling strategy by design: QMP is a simple, off-policy, and hyperparameter-free plug-in that complements other MTRL methods. We explicitly clarify the scope in the paper, and show how the improvements are significant, despite the simplicity of QMP.

Scope: We have edited our paper to incorporate off-policy more explicitly, based on your feedback:

1. Introduction: “This approach offers a simple, general, and sample-efficient approach that complements existing off-policy MTRL methods.”
2. Limitation: “Since QMP only shares behaviors via off-policy data collection, it is not applicable to on-policy RL base algorithms like PPO.”

Improvements: To provide additional perspective on performance across many benchmarks, the table below compares how much prior MTRL approaches improve the final performance over the simplest Zero-MTRL baseline, which is the dotted-blue line in Figure 7.

Two key insights emerge:

• Behavior-sharing always results in non-negative gains across environments.
• Behavior-sharing is competitive with and often outperforms other sharing methods.

Moreover, in the more complex MT50 task set, we observe an improvement of +22.8% over the parameter-sharing baseline.

We appreciate your valuable time and we hope our rebuttal justifies the complementary benefits of a simple add-on QMP method across various environments and off-policy MTRL approaches. We would love to incorporate any further specific suggestions or concerns you have.

In summary:

1. We revised “off-policy” MTRL as scope (Sec 1, 8)
2. QMP shows improvements on best algorithms from Sodhani et al [1] on MT10, MT50 and all tested environments (Fig 7, 15)
3. PCGrad addresses gradient interference, and PCGrad + QMP > PCGrad-only (Figure 15)
4. QMP is complementary to a wide coverage of MTRL baselines (Section 7).

We thank the reviewer for their thoughtful feedback and positive evaluation of our work. We are glad that you appreciated QMP’s complementary design, theoretical soundness, and extensive experiments. Below, we address your question regarding computational overhead and scalability.

QMP requires evaluating Q-values across multiple task policies at each decision step, which could introduce computational overhead, particularly in settings with a large number of tasks. The paper lacks a detailed comparison of computational costs between QMP and baseline methods, which would be helpful for assessing its scalability.

Thank you for your question. Appendix H.4 and Table 3 in the paper provide a detailed computational cost analysis. To summarize:

1. Small overhead in computational time

As shown in Table 3, QMP introduces only a 1.75% time overhead on the large 50-task set in MT50, because the total runtime is dominated by the compute cost of neural network training and environment simulation speed.

2. Efficient policy and Q-function evaluation

QMP requires evaluating N policy proposals and N Q-values to sample from $ \pi^i_{mix} $. However, these evaluations are efficiently batched and parallelized.
- In QMP + Parameter-Sharing, the multihead architectures of the policy and Q-network allow all N evaluations to be performed in one single pass through each network. Among these two, the policy pass is already performed to get the action from the current task’s policy, so the only extra overhead is due to one Q-network pass.
- Without parameter-sharing, $ Q_i(s, a_j) $ evaluations can be batched over the number of actions, and the policy evaluations $ \pi_j(a | s) $, being independent, can be obtained in parallel. Our implementation batched the Q evaluations, but did not parallelize the policy evaluations, thus incurring a small computational overhead of around 2 hr.

3. Scalability

Due to the parallelizable nature of QMP’s operations, its overhead scales well even for large task sets and in combination with parameter-sharing, requiring the extra time for only a single Q-function pass.

We thank you for your feedback and hope we've addressed your comments adequately. We are happy to answer any further questions and incorporate any further suggestions.

Dear reviewer, we appreciate your time and effort to review our work. Since it is the last day of discussion, we would be grateful for any comments or suggestions in response to our rebuttal. We would be happy to incorporate any specific suggestions. Thank you.

We sincerely thank the reviewer for their thoughtful evaluation and valuable feedback. We are pleased that you appreciated the elegance and theoretical soundness of our QMP framework, as well as its consistently complementary nature to existing MTRL methods. We address your concerns below.

1. QMP adds negligible compute time

significant computational costs, especially for large task sets as evidenced by the 7+ days runtime in MT50 experiments

We respectfully clarify that this is a misunderstanding. Table 3 shows that Parameter-Sharing baseline (no QMP) on MT50 already takes 7 days, 3 hr, while QMP + Parameter-Sharing takes 7 days, 6 hr — only a 1.75% time overhead on the large 50-task set. The total runtime is dominated by the compute cost of neural network training and environment simulation speed. In comparison, the additional compute time due to QMP is nominal. With parallelization, only one additional Q-function evaluation per policy step is needed.

2. QMP does not introduce additional dependence on accurate Q-function estimation

The method heavily relies on accurate Q-function estimation to select appropriate behaviors.

This concern likely stems from the explicit $\\arg \\max$ step before selecting a policy for action. However, the QMP objective follows directly from SAC’s underlying objective, and both are equally dependent on accurate Q-function estimation:
\ $

\pi^{\text{new}} = \arg \min_{\pi' \in \Pi} \mathrm{D}_{\mathrm{KL}} \left( \pi'(\cdot \mid s_t) \Bigg\| \frac{\exp \left(\frac{1}{\alpha}Q^{\pi^{\text{old}}}(s_t, \cdot)\right)}{Z^{\pi^{\text{old}}}(s_t)} \right)
\ $

While SAC performs gradient ascent over the current Q-function (which could be inaccurate), QMP layers an additional maximization step on top. Thus, the explicit $\\arg \\max$ in QMP is simply achieving what gradient ascent aims to do for SAC, just better — i.e., maximize whatever the current Q-function is. Both rely equally on the Q-function and do not alter their dependence on Q-function accuracy. These algorithms still work because of how generalized policy iteration (GPI) (see Section 5.1 and below).

Generalized Policy Iteration (GPI): GPI converges even with initially inaccurate Q-functions, as long as iterative updates are made. As per Sutton and Barto $1$ , in GPI, “policy evaluation and policy improvement processes interact, independent of the granularity and other details of the two processes.” Theorem D.2 shows that GPI with QMP converges at least as fast as GPI without QMP, which means that no additional instability is introduced due to QMP.

This dependency may lead to suboptimal behavior selection during early training or in tasks with sparse rewards where Q-function learning is unstable.

Sparse Rewards: It is true that sparse rewards delay (soft) Q-function learning, but this affects QMP and SAC equally, wherein the actor learning would be delayed because the primary source of actor updates is gradient ascent over the Q-function. Moreover, QMP accelerates learning once Q-function updates become positive by leveraging behaviors from other tasks more effectively. This is evident in Task 3 of Multistage Reacher (Figure 13), a sparse reward task, where QMP achieves the most improvement as it shortcuts the delay in learning of its own actor, by using other task policies.

3. All tasks except MT10 were hyperparameter-free

The method requires careful tuning of hyperparameters, such as manually setting 70% task-specific policy usage in Meta-World MT10.

This is true for MT10, where 70% conservative behavior-sharing led to minor improvements. With both 50% and 70%, we observed similar performance gains and did not tune this single hyperparameter further. In all other environments, conservative behavior-sharing was unnecessary, and no hyperparameter tuning was needed, and we observed this peculiar behavior only in MT10 and reported it transparently in the paper. Notably, the larger MT50 set result does not have any hyperparameter tuning.

We thank you for your valuable time. In summary, we show:

1. compute overhead is low (Table 3)
2. QMP is reliable because of GPI (Sec 5.1)
3. no hyperparameter-tuning is required (Appendix I)
4. QMP especially outperforms baselines in conflicting tasks (Figure 13).

If we have addressed all your concerns and justified the simplicity and soundness of QMP, we kindly request you to reconsider your rating. Please let us know if we can address any specific concerns or suggestions, as we would love the opportunity to discuss.

I appreciate the authors' thorough response to the previous concerns and their transparent discussion of these limitations. The revision has significantly improved the paper's clarity and completeness.

While this is a fundamental limitation of behavior-sharing approaches rather than a weakness specific to QMP, it would be valuable for future work to quantify this relationship between behavioral similarity and expected performance gains.

Overall, I will raise my score accordingly.