Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

Thank you for your positive review. We address your questions below.

Q1: Related work on iterative pruning

We have already discussed relevant works on dynamic sparse training (DST) in RL within our Introduction and Related Work sections. In the next version, we plan to expand our coverage of more general topology evolution methods, including the papers you mentioned. Thank you for pointing out the valuable references.

Q2: The connection between gradient norm and plasticity

Your question about distinguishing between 'large' and 'stable' gradient norms is insightful and helps us clarify an important relationship that wasn't fully explained in our manuscript.

In Figure 5, we observe that large dense networks experience a rapid collapse in gradient norm early in training while performance remains poor. This premature descent into a low gradient state reflects the agent's inability to effectively learn from new experiences, directly corresponding to the slow performance improvement in these networks. By contrast, sparse networks maintain more consistent gradient signals throughout training. Therefore, it's this early declining pattern, rather than absolute magnitude, that indicates plasticity loss.

Several studies have explored the relationship between gradient norm collapse and plasticity loss, including:

• Figure 7 in The Primacy Bias in Deep Reinforcement Learning, ICML 2022
• Section 4.2 in Loss of Plasticity in Continual Deep Reinforcement Learning, CoLLAs 2023
• Figure 10 in Weight Clipping for Deep Continual and Reinforcement Learning, RLC 2024
• Figure 5 in Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning, ICLR 2024

How to precisely measure a network's plasticity loss level remains an open question, which is why current work tends to employ multiple different metrics simultaneously for a more comprehensive assessment. The lower dormant ratios, higher SRank values, and Reset diagnostic experiments in Section 4 collectively demonstrate how sparsity helps preserve network plasticity.

Your question has prompted us to reconsider our presentation. Since gradient norm has confounding relationships with training stability, and our other metrics already establish the plasticity benefits of sparse networks, we plan to move this discussion to the appendix with more detailed explanations in our revised manuscript.

Q3: Further trend of gradient norms beyond 1M steps

Your concern about continuously increasing gradient norms prompted us to extend our experiments to 2M steps on Dog Trot and Run tasks, shown in figure (anonymous link), revealing two interesting patterns:

In Dog Trot, where performance plateaus around 1M steps, gradient norms in sparse networks eventually peak and then gradually decrease, demonstrating natural stabilization.

In Dog Run, where learning continues beyond 1M steps, gradient norms maintain moderate activity, corresponding directly with ongoing performance improvements.

These patterns suggest sparse networks achieve an effective balance: they maintain sufficient gradient activity for learning without collapsing (unlike dense networks), while also not becoming unstable over time. We'll include this extended analysis in our revised manuscript's appendix to better illustrate the long-term benefits of network sparsity.

Q4: Comparison between Erdos-Renyi ratio and other layer-wise ratios

Based on sparse training research in the broader deep learning field, the advantage of ER initialization likely stems from providing more balanced information flow across network layers. By scaling sparsity proportionally to the geometric mean of input/output dimensions (rather than uniformly), ER maintains approximately equal fan-in/fan-out ratios across layers, preventing bottlenecks in both forward and backward passes.

The core purpose of our paper is to demonstrate how network sparsity as a fundamental property helps scale up DRL model size, rather than comparing different sparsification methods. We chose ER initialization because previous sparse training studies (in both DRL and supervised learning) have established its superiority over uniform initialization. To directly compare these approaches in our setting, we conducted additional experiments shown in the figure (anonymous link).

The results reveal that at lower sparsity levels (≤0.6), both initialization methods perform comparably. However, at higher sparsities (≥0.8), uniform layer-wise ratios exhibit a dramatic performance drop. These findings suggest that:

1. Sparsity itself benefits DRL scalability regardless of the specific layer-wise ratio configuration.
2. ER initialization provides greater robustness, especially at high sparsity levels, by creating a more balanced network topology that better maintains information flow.

Thank you for your thorough review and positive evaluation of our work.

It would be interesting to see these observations extended to value-based methods in discrete action spaces, such as DQN on Atari. Nonetheless, the current findings are valuable and worth sharing with the community.

We agree that extending our findings to value-based methods with discrete action spaces is crucial for demonstrating the broader applicability of network sparsity benefits. To address this, we've conducted new experiments on the Atari-100k benchmark using Data Efficient Rainbow DQN (DER) [1] as our baseline algorithm.

The experiments compare performance improvements when scaling network width to 3x the default size with varying sparsity levels (0.0, 0.4, and 0.8). Results are shown in figure (anonymous link). (Note: Due to time constraints before the first round response deadline, we've only completed experiments on 13/26 Atari tasks. We expect to update the figure with complete results within one more day.)

These results show that introducing network sparsity while scaling up model size produces similar benefits in discrete action tasks as observed in our continuous control experiments. This further validates the general effectiveness of network sparsity for scaling DRL models across different domains and algorithms.

We note that the Atari-100k low-data regime may not fully demonstrate the benefits of scaling, and more comprehensive studies with longer training (e.g., 10M environment steps) would be valuable for future work [2]. Nevertheless, these preliminary results provide additional evidence supporting our main findings about network sparsity's role in unlocking DRL scaling potential.

[1] When to use parametric models in reinforcement learning?, NeurIPS 2019

[2] In value-based deep reinforcement learning, a pruned network is a good network, ICML 2024

Thank you very much for your positive evaluation of our work.

Q: What would happen to the learning ability when you only prune the critic or actor? Does a sparse actor play a key role?

This is an excellent question about the relative importance of sparsity in different components of actor-critic architectures. We conducted additional experiments comparing four configurations: dense networks, sparse actor only, sparse critic only, and sparse for both actor and critic.

Our results shown in figure (anonymous link) clearly show that applying sparsity to both actor and critic networks yields substantially better performance than either partial approach. Interestingly, when sparsity is applied to only one component (either actor or critic), the scaling curves closely resemble those of the dense baseline.

This suggests that scaling benefits emerge from a balanced application of sparsity across the entire architecture rather than from any single component. We believe this occurs for several reasons:

1. Critics require sparsity to avoid optimization pathologies during online TD learning. As demonstrated in Section 4, scaling up dense DRL models leads to severe plasticity loss in critics and diminished value representation capacity.
2. The need for sparse actors aligns with previous findings [1,2] showing that actors are particularly sensitive to network scaling. In actor-critic methods, since actor learning depends on critic outputs, unnecessarily complex actor networks can impede performance. The specific mechanisms behind sparse actor benefits warrant further investigation.
3. The balance between actor and critic parameters is crucial. Our experiments use the SimBa architecture, which carefully established optimal component sizing [1] - in the default 4.51M parameter configuration, the critic accounts for 4.34M parameters. When scaling our networks, we proportionally increased both actor and critic sizes, already accounting for their different representational capacity requirements. By maintaining this ratio during scaling, applying sparsity to both components becomes necessary to preserve the architectural balance that makes SimBa effective.

[1] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning, ICLR 2025

[2] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, NeurIPS 2024

Q1: Limited DMC environments affecting generalizability

We acknowledge the reviewer's concern that most experiments are conducted on DMC, which might impact the generalizability of our analyses. To address this, we've conducted new experiments on the Atari-100k benchmark using Data Efficient Rainbow DQN (DER) [1] as our baseline algorithm.

The experiments compare performance improvements when scaling network width to 3x the default size with varying sparsity levels (0.0, 0.4, and 0.8). Results are shown in figure (anonymous link).

These results show that introducing network sparsity while scaling up model size produces similar benefits in discrete action tasks as observed in our continuous control experiments. This further validates the general effectiveness of network sparsity for scaling DRL models across different domains and algorithms.

We note that the Atari-100k low-data regime may not fully demonstrate the benefits of scaling, and more comprehensive studies with longer training would be valuable for future work. Nevertheless, these preliminary results provide additional evidence supporting our main findings about network sparsity's role in unlocking DRL scaling potential.

[1] When to use parametric models in reinforcement learning?, NeurIPS 2019

Q2: Fixed uniform sparsity ratios vs. Erdos-Renyi ratios

Based on the reviewer's question, we conducted new experiments comparing the naive approach of fixed/uniform sparsity ratios across all layers with the Erdos-Renyi (ER) ratio. The results shown in the figure (anonymous link) reveal that at lower sparsity levels (≤0.6), both initialization methods perform comparably, while at higher sparsities (≥0.8), uniform layer-wise ratios exhibit a dramatic performance drop.

The advantage of ER initialization stems from its ability to provide more balanced information flow across network layers. By scaling sparsity proportionally to the geometric mean of input/output dimensions, ER maintains approximately equal fan-in/fan-out ratios across layers, preventing bottlenecks that emerge with uniform sparsity at high sparsity levels where network connectivity becomes critical for information propagation.

We want to emphasize that our paper's focus is on demonstrating how network sparsity as a fundamental property enables DRL model scaling, rather than identifying the optimal sparse topology. The fact that uniform sparsity at appropriate levels also improves performance further supports our main claim that sparsity itself (regardless of specific implementation) is a key enabler for unlocking the scaling potential of DRL networks.

Q3: Random sparsity benefits vs. Lottery Ticket Hypothesis findings

The reviewer raises an important question about reconciling our findings with the Lottery Ticket Hypothesis (LTH). The key difference lies in the fundamentally different problem settings and objectives:

First, we must distinguish between previous sparse training studies and our work. Earlier studies on sparse training (including LTH) were motivated by model compression to reduce computational costs while maintaining performance. Such approaches operate under the assumption that in supervised or unsupervised learning, larger dense networks generally yield better performance—the core premise behind modern scaling laws.

However, this assumption is violated in online DRL settings. As demonstrated in Figure 1 of our manuscript, online DRL networks face severe scaling barriers where increasing model size not only fails to improve performance but often leads to catastrophic collapse. Our analysis in Section 4 further reveals that these scaling barriers emerge because larger dense networks are more susceptible to optimization pathologies during online RL training, preventing them from leveraging their theoretical capacity.

Our work aims to demonstrate that network sparsity itself, as a fundamental network property, can mitigate these pathologies and unlock the scaling potential of DRL networks. This represents a fundamentally different objective than LTH's focus on finding efficient subnetworks within larger models.

Therefore, our findings don't contradict the Lottery Ticket Hypothesis but rather highlight the unique utility of network sparsity in addressing online DRL's specific pathologies and inscalability. For a given sparsity level, we believe that better topologies (such as "winning tickets") could outperform one-shot random pruning. However, finding tickets requires significant computational resources, and discovering "lottery tickets" without training remains challenging. Efficiently identifying winning tickets as trainable networks represents valuable future work, but identifying optimal sparse topologies was not the focus of our current study.