MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning

Q1: Evaluation on Multi-task Learning:

We appreciate the reviewer’s suggestion regarding multi-task evaluation. We have conducted additional experiments where our method is trained jointly on Meta-World tasks using a single policy. The performance is compared against MLP baseline under the same multi-task setup. Experimental details and results are provided on the rebuttal Section A.

Q2: Cut-off Frames of different tasks:

Thank you for your comment. Our frame settings follow prior works such as DrQv2 [14], TACO [15], and DrM [16], and are chosen to reflect task difficulty while maintaining consistency in comparison. We agree that cut-off frames should account for whether baselines have converged. As noted, our main comparisons are against DrM [16], which consistently converges by the reported cut-off in most tasks.

Q3: “While the design of two main contributions makes sense separately, I feel that they are contradicting each other.”:

Thanks for pointing out your confusion. We will explain why these two main contributions do not contradict each other.

For MoE: we would like to firstly clarify the different concepts of agent and expert. In our context, agent refers to the policy agent, whose backbone is a MoE, and MoE consists of several experts. So the assumption of MoE in your original comment — “different agents will master different skills to avoid gradient conflicts” — should be rephrased as “different experts will master different skills to avoid gradient conflicts”.

For Task-oriented Perturbation Mechanism: During RL training, we maintain a fixed-size set $S_{\text{top}} = \{(\theta, R)\}$, which stores the weights $\theta$ and corresponding episode rewards $R$ of the top-performing agents seen so far. At each episode $t$, if the current agent’s reward $R_t$ exceeds the lowest reward in $S_{\text{top}}$, we replace the corresponding tuple with $(\theta_t, R_t)$.

Suppose now we have $N$ history top-performing agent weights (each has $M$ experts) in $S_{\text{top}}$ and we want to execute task-oriented perturbation, the goal distribution to perturbing $\mathrm{expert}_{i}$ is $\mathcal{N}(\mu_i, \sigma_i)$, where $\mu_i$ and $\sigma_i$ are the mean and std of

{exper$\text t_i^{\theta_k}$ $\mid \theta_k \in S_{\text{top}}$},

in these $|S_{\mathrm{top}}|$ agents, and has nothing to do with other experts. So the perturbation process will not make experts similar but will further diversify them from each other.

We hope this is sufficient to clarify that these two designs are not contradictory. More details could be found in the original submission Section 3.2 and feel free to ask if you have more questions!

Q4: Generalization and Robustness of MoE Design:

We understand the reviewer’s concern regarding the robustness of the MoE design to hyperparameter choices. To this end, we conducted experiments on the Hammer (Sparse) task, as shown in the rebuttal Section B (due to the time limitations, we only report the ablation study on this task), varying the number of experts and top_k. Results show that while the optimal setting is MoE has 8 experts, performance remains consistent across 4, 8, and 32 experts as long as top_k = 4. This suggests the model is not overly sensitive to the number of experts.

---

[14] Yarats, D.,et.al. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645.

[15] Zheng, R., Wang, X., Sun, Y., Ma, S., Zhao, J., Xu, H., ... & Huang, F. (2023). TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning. Advances in Neural Information Processing Systems, 36, 48203-48225.

[16] Xu, G.,et.al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668.

Q1: “The impact of internal parameter changes on the overall performance may not have been further analyzed in depth… Additionally, there were no relevant ablation experiments regarding the perturbation factor.”:

We appreciate the reviewer’s attention to this point. Due to time limitations, we only report the ablation study for the number of experts and top_k in the Hammer (Sparse) task, as shown in the rebuttal Section B. The results indicate that in the Hammer (Sparse) task, the optimal choice for the number of experts is 8 and for top_k is 4. When top_k is 4, there are no significant performance differences when the number of experts is set to 4, 8, or 32, which suggests that 4 experts are enough to learn the skill in this task. The ablation on top_k further validates our hypothesis, as reducing top_k (to 2) results in a worse learning curve. If the number of experts is set to 1 and top_k is also 1, the MoE will downgrade to a standard MLP, resulting in the worst performance among all configurations. Regarding the perturbation hyperparameters, we follow DrM [11] by adopting the same perturbation factor and α values from the open-source github repo.

Q2: “Whether the MoE architecture and the task-oriented perturbation mechanism increase the computational complexity of the algorithm.”:

We agree that computational efficiency is an important concern. Importantly, both MoE and the perturbation mechanism do not increase the theoretical computational complexity of the algorithm. Empirically, as noted in the original submission Appendix E: Time Efficiency of MENTOR, while MoE may introduce higher latency compared to standard MLPs, this is primarily due to hardware-level optimizations (e.g., memory access patterns), rather than additional computation. Recent developments such as MoE-Infinity [12], which demonstrate the potential to reduce MoE latency to negligible levels.

Q3: “Insufficient in-depth theoretical analysis and mathematical proof”:

We thank the reviewer for the insightful comment. We acknowledge the importance of a thorough theoretical analysis of how the MoE architecture and task-oriented perturbation help reduce gradient conflicts; however, this is beyond the scope of the present work. Concurrent works also investigate similar patterns through an empirical manner, for example, STGC [13] also applied dense experiments to show the use of MoE to mitigate gradient conflicts for NLP tasks.

---

[11] Xu, G.,et.al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668.

[12] Fu, Y.,et.al. MoE-CAP: Cost-Accuracy-Performance Benchmarking for Mixture-of-Experts Systems. arXiv preprint arXiv:2412.07067.

[13] Yang, L.,et.al. Solving token gradient conflict in mixture-of-experts for large vision-language model. arXiv preprint arXiv:2406.19905.

Q1: Regarding the use of more recent benchmarks such as LIBERO that emphasize large-scale multi-task and lifelong learning:

We appreciate the reviewer’s suggestion of more recent benchmarks, like life-long learning tasks. We consider this a valuable avenue for future work. However, we note that this is beyond the current scope of our work, which focuses on evaluating our approach under widely used and established benchmarks and does not consider learning throughout lifetime. Recent works such as Streaming RL [5] and Ace [6] also primarily evaluate on the same or similar settings. Incorporating benchmarks like LIBERO remains an exciting extension for future research direction.

Q2: Regarding the specificity of the MoE architecture to visual RL:

We agree with the reviewer that the MoE architecture is versatile and not tied to visual RL. In this work, we focus on visual RL because of its growing attention in research ([6][7][8]) and importance in real-world applications, where agents often rely on visual input. However, we believe that MENTOR is also applicable to state-based RL tasks. To support this, we conducted a fully state-based RL training experiment using Humanoid-Gym [9], where an agent must control a humanoid to accomplish a locomotion task. The comparison results, shown on the rebuttal Section C, indicate the effectiveness of our method.

Q3: Regarding the computational overhead of MoE layers:

We thank the reviewer for pointing this out. We have discussed this in the original submission Appendix E: Time Efficiency of MENTOR, while MoE generally improves sample efficiency, it does introduce higher latency compared to standard MLPs due to its routing and expert selection. We acknowledge this tradeoff, but we want to highlight that the latency mostly comes from hardware optimization, not more computation overhead. Recent advances such as MoE-Infinity [10] offer promising directions to significantly reduce this overhead, potentially narrowing the gap to a negligible level.

---

[5] Elsayed, M.,et.al. Streaming Deep Reinforcement Learning Finally Works. arXiv preprint arXiv:2410.14606.

[6] Ji, T., Liang,et.al. Ace: Off-policy actor-critic with causality-aware entropy regularization. arXiv preprint arXiv:2402.14528.

[7] Hafner, D.,et.al. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603.

[8] Laskin, M.,et.al. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning (pp. 5639-5650). PMLR.

[9] Gu, X.,et.al. Humanoid-gym: Reinforcement learning for humanoid robot with zero-shot sim2real transfer. arXiv preprint arXiv:2404.05695.

[10] Fu, Y.,et.al. MoE-CAP: Cost-Accuracy-Performance Benchmarking for Mixture-of-Experts Systems. arXiv preprint arXiv:2412.07067.

Q1: Regarding the weakness of real-world RL:

We agree that real-world RL still faces several limitations. However, we have seen progress in this area. Recent work, such as Serl [1], has released a software suite that facilitates the rapid deployment of real-world learning paradigms. Moreover, recent advances in foundation models [2][3][4] can help simplify the design of perception systems for computing object-relevant rewards. In addition, we would like to emphasize that the main scope of our work is to propose a sample-efficient RL algorithm, which is not limited to real-world RL scenarios.

Q2: Figure 5 is missing error bars?

Thanks for your comment, we will add it to the final camera-ready version.

Q3: In the real world tasks are there any multi-task setups?

We trained three separate policies for three real-world tasks, each featuring specific challenges (details are provided in Section 4.2). In particular, the peg insertion task is designed to evaluate multi-task learning capability, as the policy must learn to insert different pegs into corresponding target holes using a single visual policy. We visualized the expert usage heatmap of MENTOR for this task in the original submission Appendix G, which shows that the MoE agent tends to utilize different experts for different plug shapes. The ablation study results shown in the original submission Table 1 also demonstrate the effectiveness of the MoE structure compared to a standard MLP.

Q4: If the experiments in Figure 6 really leverage MOE? Can you show more multi-task experiments?

Generally speaking, MENTOR is designed to improve the sample efficiency of visual RL, rather than focusing purely on multi-task scenarios. We found that MoE can promote learning efficiency by automatically leveraging distinct experts to decompose a hard task into several sub-tasks and learn them separately. Therefore, the experiments in Figure 6 are trained with a single policy per task. We really appreciate your valuable advice, so we conducted several multi-task learning experiments (detailed multi-task setup information is on rebuttal Section A) to demonstrate MoE’s contribution in multi-task setups.

Q5: Explanation of Task-oriented Perturbation Mechanism:

$\Phi_{\text{oriented}}$ is an approximate distribution over a set of high-performing agent weights $S_{\text{top}}$, which is constructed as training processes. Specifically, we maintain a fixed-size set $S_{\text{top}} = \{(\theta, R)\}$, which stores the weights $\theta$ and corresponding episode rewards $R$ of the top-performing agents seen so far. At each episode $t$, if the current agent’s reward $R_t$ exceeds the lowest reward in $S_{\text{top}}$, we replace the corresponding tuple with $(\theta_t, R_t)$. For task-oriented perturbation, we approximate $\Phi_{\text{oriented}}$ as a Gaussian $\mathcal{N}(\mu_\theta^{\text{top}}, \sigma_\theta^{\text{top}})$, where the mean and standard deviation are computed over the network weights in $S_{\text{top}}$. This dynamic update ensures that $\Phi_{\text{oriented}}$ remains focused on the most promising regions of the parameter space and generates more effective perturbation candidates $\phi$ compared with perturbation using random weights. A detailed description could be found in the original paper Section 3.2.

Q6: “Why specifically experts 9, 13, 14, and 15 are selected for visualization…”:

The MoE agent used in this task (Meta-World Assembly) has 16 experts, with the top_k parameter set to 4. Therefore, in Figure 4, we mainly visualize the 4 most active experts during task execution, which happen to be experts 9, 13, 14, and 15. This selection may vary depending on the environment's random seed and the random initialization of expert weights. After training, the agent primarily uses these 4 experts, while the others exhibit low utilization during execution, so we did not include them in the visualization.

---

[1] Luo, J.,et.al. Serl: A software suite for sample-efficient robotic reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 16961-16969). IEEE.

[2] Liu, S.,et.al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision (pp. 38-55). Cham: Springer Nature Switzerland.

[3] Wen, B.,et.al. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17868-17879).

[4] Wen, B.,et.al. FoundationStereo: Zero-Shot Stereo Matching. arXiv preprint arXiv:2501.09898.