ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning

We thank Reviewer Cp7C for their thoughtful feedback and valuable suggestions, which have significantly improved our paper. We have carefully addressed each comment, strengthened our experimental results, and clarified our key contributions accordingly. Below, we respond in detail to each point raised by the reviewer.

Other Multi-Task RL Baseline (PopArt and MOORE):

We appreciate the reviewer’s suggestion regarding the inclusion of an additional multi-task RL baseline, specifically PopArt[1] and MOORE [1]. In response, we conducted comparative experiments using MOORE on the MT10 and MT50 benchmarks.

Since the Meta-World v2 environments default to a horizon of 500, we used this setting across all our experiments. However, the original MOORE paper used a horizon of 150, making a direct comparison with our ARS results challenging. To address this, we attempted to run MOORE with a horizon of 500, but the official code required roughly 200 hours of computation on the MT50 benchmark alone. Consequently, we evaluated ARS using a horizon length of 150 for fairness. The ARS results are shown in Table 1 at the following anonymous link:

https://sites.google.com/view/icml25ars

Notably, the MOORE setup on MT50 (n_expert=6) includes significantly more parameters than our default ARS (400×4). To ensure a fairer comparison, we tested MOORE against an enlarged ARS variant (800×4), which has a comparable parameter count. Despite the comparable model size, ARS consistently outperformed MOORE on both benchmarks requiring significantly less computational time. For instance, on MT50, MOORE requires about 200 hours to train, whereas ARS (800×4) surpasses its performance in just 22 hours—an order-of-magnitude faster. These results demonstrate that our ARS framework achieves significant performance improvements without incurring substantial computational overhead. We will include MOORE results in the revised version.

We also tested PopArt on MT10. Because the official implementation is unavailable, we reimplemented PopArt’s scale-invariant updates within SAC-MT, using SAC target values for critic learning. We varied the update frequency over {1, 10, 100, 500}. The outcomes are shown in Table 2 (available anonymously at https://sites.google.com/view/icml25ars). Across all frequencies, our ARS method outperformed PopArt on MT10, and we plan to include the complete results for both MT10 and MT50 in our revised paper.

https://sites.google.com/view/icml25ars

Our ARS method consistently outperformed all PopArt variants on the MT10 benchmark. We will include comprehensive PopArt variant results for both the MT10 and MT50 benchmarks in the revised paper.

[1] Hessel, Matteo, et al. "Multi-task deep reinforcement learning with popart." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

[2] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. "Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts." The Twelfth International Conference on Learning Representations.

Effectiveness of Reset Mechansim

We thank the reviewer for suggesting a more thorough analysis of the Reset Mechanism.Table 4 in our paper already highlights its importance in stabilizing performance. To further investigate, we analyzed Q-values per task during training on MT10, comparing ARS with and without the reset mechanism. Results are shown in Figure 1 at the following anonymous link:

https://sites.google.com/view/icml25ars

As shown in Figure 1, training without the reset mechanism leads to significantly lower Q-values across tasks, with negative values appearing in tasks like 'push' and 'pick-place'—despite rewards being always positive. Q-values also show greater variance without reset, emphasizing the reset mechanism’s role in ensuring training stability.

Random Goal Positions

Sorry for the confusion. Random goal positions are used in our setup.

Lower Performance of PaCo in MT10

We use the same hyperparameters for PaCo as in the original paper. The performance gap likely stems from the difference in horizon lengths: the original PaCo uses 150, while Meta-World v2 defaults to 500, which we follow. To isolate this effect, we also run ARS with a horizon of 150. Results are shown in Table 1 at the anonymous link below:

https://sites.google.com/view/icml25ars

ESTR Results

We sincerely apologize for the confusion regarding the ESTR results. The reported ESTR value of PaCo at $\delta = 0.7$ was incorrect and should be revised from 0.6 to 0.5.

Definition of H

Yes, H is the horizon length.

Frequency (n_reset) for ARS w/o reset

We investigated the performance of the ARS w/o reset baseline with various values of n_reset . We then selected the best-performing setting for the paper. The value of n_reset used in the paper for the ARS w/o reset baseline is 40.

We thank Reviewer mXD6 for their thoughtful feedback and valuable suggestions, which have significantly improved our paper. We have carefully addressed each comment, strengthened our experimental results, and clarified our key contributions accordingly. Below, we respond in detail to each point raised by the reviewer

Limited Environments

We appreciate the reviewer’s suggestion of three benchmarks for multi-task RL. We believe that the MT10 and MT50 benchmarks from Meta-World are widely recognized and present significant challenges in multi-task RL. Since no existing method has fully solved both, demonstrating strong performance on these benchmarks effectively highlights the strength of our approach.

That said, we agree that including the suggested benchmarks could further strengthen our claims. Benchmarks [2] and [3] are primarily designed for demonstration-driven settings, such as imitation learning and offline RL. While our current focus is on the online setting, we believe that extending our ARS method to offline multi-task RL would be an interesting direction for future work. We appreciate the reviewer for introducing these valuable benchmarks.

Reward Scale Issue with Sparse Reward Setting

Even in sparse reward settings, task difficulty can vary significantly across tasks, leading to large variations in returns during training. This can cause an uneven reward distribution in the replay buffer. Applying the ARS framework can help mitigate this issue.

Instability caused by varying reward scales

Large variations in reward scales during training can indeed destabilize the learning process. To mitigate this, we update the reward scales only during reset periods—four times in MT10 and six times in MT50. This reset mechanism helps maintain stability by preventing frequent changes in reward scaling. Additionally, since the networks are initially trained with the established reward scales, this approach further supports stable training throughout.

We thank Reviewer SzVus for their thoughtful feedback and valuable suggestions, which have significantly improved our paper. We have carefully addressed each comment, strengthened our experimental results, and clarified our key contributions accordingly. Below, we respond in detail to each point raised by the reviewer

Other Multi-Task RL Baseline (MOORE [1]):

We appreciate the reviewer’s suggestion to include an additional multi-task RL baseline, specifically MOORE [1]. In response, we conducted comparative experiments using MOORE on the MT10 and MT50 benchmarks.

In the default setup of the MetaWorld v2 environments, the horizon length is set to 500, which was used consistently across all our experiments. However, we noticed that the experiments reported in the original MOORE paper [1] were conducted using a horizon length of 150, making a direct comparison between the reported MOORE results and our ARS method inappropriate.

To address this, we attempted to run MOORE with the horizon length set to 500. Unfortunately, we encountered significant computational challenges, as the official implementation requires approximately 200 hours to complete on the MT50 benchmark. To ensure fairness, we evaluated ARS using a horizon length of 150. The ARS results are shown in Table 1 at the following anonymous link:

https://sites.google.com/view/icml25ars

The MOORE setup (n_expert = 6) on MT50 uses significantly more parameters than our default ARS (400×4). To ensure a fair comparison, we evaluated MOORE against a larger ARS variant (800×4) with a similar number of parameters. Despite the comparable model size, ARS consistently outperformed MOORE on both benchmarks while also requiring dramatically lower computational costs. Specifically, on MT50, MOORE requires approximately 200 hours of training, while ARS (800×4) achieves superior performance in just 22 hours—an order-of-magnitude reduction in training time. These results demonstrate that our ARS framework achieves significant performance improvements without incurring substantial computational overhead. We will include MOORE results in the final version.

[1] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. "Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts." The Twelfth International Conference on Learning Representations.

Discussion of Recent Advances in Reward Normalization and Adaptive Scaling across Different RL Domains .

We thank the reviewer for recommending additional related work. We will expand our related work section accordingly to enhance the paper.

Evaluation is Limited to Meta-World Benchmarks

We acknowledge the importance of including experiments beyond Meta-World. While the MT10 and MT50 benchmarks are widely recognized and challenging in multi-task RL—with no method fully solving both—they serve as a strong demonstration of our approach's effectiveness. Nevertheless, we welcome suggestions for additional benchmarks and will gladly conduct further experiments to strengthen our claims.

Lower Performance of Soft Modular with ARS in MT10

We attribute the initially lower performance of Soft Modular with ARS on MT10 to suboptimal hyperparameter tuning. By increasing the batch size per task from 100 to 128, we significantly improved performance to $98.8 \pm 1.3$.

Computational Overhead and Potential Limitations Introduced by the Reset Mechanism

The reset mechanism introduces minimal computational overhead, occurring only during reset periods. With our reset strategy (updating re-initialized networks 1,000 times per reset), the total number of updates increases only slightly from 2,000,000 to 2,006,000—an increase of just 0.3%.

How can ARS be extended or adapted to other multi-task or multi-agent settings?

As illustrated in Algorithm 1 and Figure 14, the ARS framework is easily adaptable to any off-policy multi-task RL method by incorporating an adaptive reward scaling factor based on the replay buffer.

In multi-agent settings, we can consider either individual rewards per agent or a global reward. With a global reward, applying ARS is challenging since it effectively becomes a single task. In contrast, individual rewards allow each agent to be treated as a separate task, making ARS more applicable. However, caution is needed—unlike multi-task learning where tasks are independent, agents in multi-agent systems are interdependent. Thus, careful design of the adaptive reward scaling factor is crucial.

Limited Theoretical Analysis

We agree with the reviewer that the theoretical analysis is limited. However, we believe the extensive experimental validation provides strong support for the effectiveness of ARS. We plan to address a comprehensive theoretical analysis in future work.

Other Comments Or Suggestions:

We thank the reviewer for identifying typos, issues with bold symbols, and citation errors. These will be corrected thoroughly in the revised paper.