We thank the reviewer for their feedback! We are happy that the reviewer found that “the paper is well written and easy to follow”, and that it “provides extensive empirical evaluations”. We are also glad the reviewer recognized that the "interventions are simple and easy to implement, making it easy for others to adapt”. We respond to their main concerns below.

As a nitpick, while overall the plots are very nice, the readability should be improved by properly scaling the text to the template. I find the text to be relatively small in many of the plots.

Thank you for pointing this out. We agree that readability is important and will ensure that all figures are properly scaled to meet the template guidelines in the final version.

I believe the authors are missing a few references in their related work ..

Thank you for pointing us to these relevant works. We agree that they are highly relevant to the discussion on scaling in deep RL and will include them in the final version. While these works focus on batch/weight normalization and architectural constraints, our method offers a complementary and distinct approach by identifying multi-skip connectivity and Kronecker-factored optimization as synergistic mechanisms that improve both gradient propagation and parameter efficiency. In the revised manuscript, we will discuss these papers as approaches that analyze optimization pathologies in the context of scaling and propose architectural interventions to stabilize deep RL training.

Why did the authors specifically choose these two interventions? Did they also try other interventions to improve the gradient pathologies?

Thank you for raising this important point. We selected Kronecker-factored optimization (Kron) and multi-skip connectivity based on the outcome of a systematic exploration of several candidate interventions aimed at improving gradient flow, plasticity, and stability in deep RL.

As part of our investigation, we evaluated a wide range of alternative methods from prior work that aim to mitigate optimization pathologies in deep RL [2,10,11]. The methods were chosen because they have been used in other domains (e.g., supervised learning) and also deep RL to alleviate similar gradient-related issues. We excluded ReDo [1], Pruning [2], and Reset [3] as they require extensive hyperparameter tuning and, based on [2], their performance collapsed or stagnated when increasing network capacity. The evaluated methods include:

• Second-order and adaptive optimizers: Apollo [4], Shampoo [5], AdaBelief [6]

• Regularizers: L2 norm penalties [7], weight clipping [8], weight decay

• Activation functions: GELU, CReLU [9]

• Learning rate schedules: Cosine annealing and cyclic schedulers.

• Learning rates at different scales: we try both multiplying and dividing the default learning rate (2.5e-4) by 12 to potentially compensate for the increased networks scale.

As summarized in the table below (3 seeds, mean performance), none of these interventions consistently showed improved performance compared to our proposed combination.

Particularly for second order optimizers, one of our main interventions, we explored Apollo, Shampoo, and AdaBelief, as well as different architectural variants like ResNets and DenseNets. These results are included in the appendix (see Figure 13 for optimizer comparisons and Section D.3). Despite extensive hyperparameter tuning across both architectures and optimizer, these alternatives did not consistently improve trainability in deep RL.

In contrast, our proposed combination of Kronecker-factored optimization (Kron) and multi-skip connections consistently outperformed these baselines. To strengthen this claim, we ran additional experiments for the rebuttal on the Atari-10 benchmark [12], comparing skip connectivity patterns when paired with Kron. Results show that richer skip connectivity is generally beneficial, with our multi-skip design performing best:

Table: Human-normalized scores on Atari-10 [12]. More skip connections generally improve performance, with multi-skip connections yielding the best results.

We will add a brief discussion in the paper clarifying that these interventions were selected based on systematic experimentation, as documented in the appendix.

Could the authors clarify which confidence intervals were used for the plots in the main part of the paper?

Thank you for the question. All plots in the main paper for deep RL experiments report 95% confidence intervals based on 3 independent seeds, following standard practice in recent deep RL literature [13]. We will clarify this in the figure captions and main text in the final version to avoid ambiguity.

In addition to per-task learning curves, we report aggregate performance metrics across tasks, including interquartile mean (IQM), median [13]. This aggregate result provides a more reliable estimate of overall performance and demonstrates that our improvements are statistically significant and consistent across environments.

---

Thank you for the thorough review. Please let us know if any points remain unclear.

References

[1]. JSO Ceron et al, In value-based deep reinforcement learning, a pruned network is a good network , ICML’24

[2]. G Sokar et al, The dormant neuron phenomenon in deep reinforcement learning, ICML’23

[3]. E Nikishin et al, The primacy bias in deep reinforcement learning, ICML’22

[4]. Xuezhe, Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization, arXiv’21

[5]. Vineet Gupta et al, Shampoo: Preconditioned Stochastic Tensor Optimization, arXiv’18

[6]. Juntang Zhuang et al, Adabelief optimizer: Adapting stepsizes by the belief in observed gradients, NeurIPS’20

[7]. A Kumar et al, Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes, ICLR’23

[8]. Mohamed Elsaye et al, Weight clipping for deep continual and reinforcement learning, RLC’24

[9]. Zaheer et al, Loss of Plasticity in Continual Deep Reinforcement Learning, CoLLAs’23

[10]. Skander et al, No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO, NeurIPS’24

[11]. Arthur et al, A Study of Plasticity Loss in On-Policy Deep Reinforcement Learnin, NeurIPS’24

[12]. Aitchison, Matthew, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. ICML’23.

[13]. R Agarwal, M Schwarzer, PS Castro, A Courville, MG Bellemare, Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS’21

We thank the reviewer for their feedback! We are happy that the reviewer found that “the paper is clearly written” and that “the results are consistent across many different architecture types and algorithms”. We respond to their main concerns below.

The novelty is not that high as skip connections are well-known, the Kron optimizer is an existing one, and gradient/plasticity issues are discussed in other works as well.

While the individual components (skip connections, Kronecker-factored optimization, and plasticity analysis) are based on existing ideas, our contributions lie in how we integrate and apply them in a deep RL context to address specific failure modes, particularly those involving gradient degradation.

While Kronecker-factored methods (e.g., [1]) have been explored before, they are not commonly adopted in deep RL. Our contribution includes a careful and extensive empirical study of using a Kronecker-factored optimizer within modern deep RL. We position this as a step toward broadening the toolkit for deep RL practitioners.

For architectures, we extend prior work on skip connections with our multi-skip network, which broadcasts CNN-extracted features to all subsequent layers. To our knowledge, this form of broadcasting has not been explored in deep RL, making the design novel. Figure 4 illustrates its structure relative to standard ResNet-style MLPs and DenseNets. Importantly, these multi-skip connections emerged as an effective mechanism for mitigating the vanishing gradient pathologies identified in our analysis. We have updated the paper to make the novelty clear.

The gradient magnitudes becoming smaller may be coincidental to the performance dropping; it did not seem proven that performance drops because the gradient magnitudes are small.

While asserting a direct causal relationship is difficult, we do observe a strong correlation between vanishing gradients and performance drop. The fact that performance increases when we mitigate vanishing gradients strengthens this connection. Nonetheless, we will clarify in the final version that this correlation does not explicitly imply causality.

In Figure 5 (right), is the gradient magnitude for +Kron the magnitude of the update vector (i.e., inverse curvature * gradient), or is it just the magnitude of the gradient?

It represents the magnitude of the preconditioned gradient. That is, the update vector resulting from the multiplication of the gradient by the preconditioning matrix.

To clarify, does the optimizer lead to different regions in the optimization space where the gradient does not become small, or does it just add a preconditioner to the gradient that adjusts the magnitude?

This is a great question, but quite difficult to answer assertively, particularly given the high levels of non-stationarity in deep RL (which result in shifting optimization landscapes). We hypothesize that the improvements we observe are primarily due to better pre-conditioning, though a deeper understanding remains an open question. We agree this is an important direction for future investigation and we will add a note in the discussion accordingly.

---

Thank you for the thorough review. Please let us know if any points remain unclear.

References

[1]. Firouzi, Mohammad. KF-LAX: Kronecker-factored curvature estimation for control variate optimization in reinforcement learning. arXiv’18.

We sincerely appreciate the reviewer’s valuable comments. We are happy that the reviewer found that “the paper is well-written and clear”. We're also pleased that the reviewer recognized “the systematic nature of our study” and that we present results “on several environments from the ALE environment and Issac Gym for a number of network architectures and different RL algorithms”. We respond to their main concerns below.

Neither of the proposed methods to stabilize the performance is novel and has indeed been studied extensively in several other contexts where the training dynamics were stable.

We agree that the individual techniques we employ, such as architectural modifications and second-order optimization, have been explored in other domains with more stable training dynamics. Our contribution lies in demonstrating how these techniques, when adapted thoughtfully, can address fundamental challenges in deep RL: the implicit non-stationarity that undermines plasticity and learning stability; a problem that is further exacerbated as model scale increases [1,2,3]. We approach this through targeted gradient interventions, which we show to be highly effective across diverse tasks and rl algorithms.

While some components are well-known individually, our multi-skip architecture, which broadcasts CNN features to all downstream layers (see Figure 4), proved particularly effective for improving gradient flow under non-stationarity conditions. We believe the thoroughness of our experimentation and analysis makes these findings a valuable foundation for future research in deep RL at scale.

Of the 2 proposals, I believe usage of higher-order optimizers to be the least novel innovation. I would assume in most contexts, integrating higher-order information would improve training ...

We agree that incorporating higher-order information should improve training in principle, and that computational cost remains a key barrier. However, in deep RL, simply introducing second-order methods does not guarantee improvements. In fact, our experiments with other second-order optimizers such as Shampoo [4] and Apollo [5], despite extensive hyperparameter tuning, did not yield gains (see Figure 13 in the appendix). This suggests that applying second-order optimization in deep RL is non-trivial.

Our analyses and systematic experimentation were key in identifying Kronecker-factored optimization as particularly effective for improving trainability under non-stationary settings. We believe this careful empirical investigation offers valuable insights for future work aiming to move beyond the standard reliance on Adam in deep RL and towards scaling deep RL.

Skip connections were studied extensively in the context of CNNs, and it's not clear from the study what the minimum the skip connections should be to ensure gradient information is passed effectively. How does the change in the skip length impact the training dynamics?

We appreciate the reviewer’s insightful comment regarding the importance of understanding the minimum number and placement of skip connections required to ensure effective gradient propagation. While our original submission did not include a detailed ablation on this point, we have now conducted additional experiments to specifically probe this question.

To isolate the effect of skip length on performance, we fix the main network of our proposed MultiSkip architecture (in large size, which includes 5 residual blocks) and vary how many of these blocks receive skip connections from the encoder. When Skip = k, we apply the encoder features as skip connections to the first k residual blocks immediately following the encoder, while the remaining (5 - k) blocks operate without direct encoder input. The table below reports human-normalized scores on the Atari-10 benchmark [6]:

Performance steadily improves as more skip connections are added, peaking when all 5 blocks are connected. This supports our original design decision to broadcast features to all MultiSkip blocks.

Furthermore, our main paper already demonstrates consistent gains when skip connections are present in the CNN network. In particular, Figure 9 reports large improvements when using the Impala CNN architecture (which includes residual connections) compared to a plain CNN.

---

Thank you for the thoughtful review. We hope these additions address the main concerns regarding novelty and applicability. Please let us know if any points remain unclear.

References

[1]. Guozheng Ma et at, Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning, ICML’25

[2]. JSO Ceron et al, In value-based deep reinforcement learning, a pruned network is a good network, ICML’24

[3]. JSO Ceron et al, Mixtures of Experts Unlock Parameter Scaling for Deep RL, ICML’24

[4]. Vineet Gupta et al, Shampoo: Preconditioned Stochastic Tensor Optimization, arXiv’18

[5]. Xuezhe, Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization, arXiv’21

[6]. Aitchison, Matthew, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. ICML’23.

We thank the reviewer for their feedback! We are happy that the reviewer found that “the paper is well-written and clear” and that “the experiments to study this problem are extensive”. We respond to their main concerns below.

Q1) Is it true that only MLP’s (aside from convnet image encoders for ALE) were studied in this paper?

Yes. Our experiments focus on combinations of convolutional backbones (e.g. to process ALE frames) and fully‑connected heads i.e., MLP architectures, mirroring the vast majority of prior model‑free deep RL work [1,2,3].

Q2) How would you apply these approaches to different architectures (transformers, diffusion policy, etc)? Or is more research needed?

Kron and standard normalization layers extend directly to any layer whose parameters or activations admit a preconditioning or normalization step: transformers, LSTMs, diffusion‑policy networks, and others.

Our multi‑skip broadcast mechanism relies on propagating intermediate feature maps forward to all downstream layers. While the concept could extend to attention-based models (e.g. broadcasting early Transformer block outputs to later ones), deciding what to broadcast (keys, values, token embeddings, etc.) and how to merge them meaningfully (residual, cross-attention, etc.) is non-trivial. We expect this to require additional research to identify the right injection and aggregation strategies in attention‑based or recurrent settings. We view this as promising future work, and we will add a discussion around this in the final version.

We also ran PQN with LSTM policies on the full ALE benchmark (57 games, 200M frames) to evaluate whether our gradient interventions generalize to recurrent architectures. The table below shows improvements of Kron + MultiSkip + LSTM over Adam + MLP + LSTM across all environments.

Due to space constraints, we cannot include the full per-game results in the main text or in the rebuttal, but we will add them to the appendix, along with:

• Per-game learning curves (similar to Figs. 16 and 17)

• Per-game % improvements (similar to Fig. 6, right).

Q3) Can the approach be augmented for different rates of change in stationarity? How much non-stationarity can the new approach tolerate before failure?

Yes, our approach generalizes across a range of non-stationarity regimes, and our results suggest it performs robustly under varying degrees of stationarity.

Previously, we demonstrated this in online RL experiments, where non-stationarity arises primarily from policy-induced distribution shifts and target bootstrapping. To further evaluate the generality of our method across different degrees of non-stationarity, we introduced new offline RL experiments where the only form of non-stationarity comes from bootstrap targets (i.e., no policy-induced drift, but still changes in the network targets for actions after Bellman backups propagate), using a challenging benchmark from a recent study [4]. Park, Seohong, et al. (2025) showed that standard offline Q-learning baselines fail even with 100M samples and near-perfect coverage.

We retrained those same baselines with and without our Kron + Multi-Skip interventions. Across nearly all environments and algorithms, our method consistently improves performance, even in these highly brittle offline settings (see table below). Notably, in difficult environments like humanoidmaze-giant-navigate, gains were substantial (in many cases even over 90%).

Baseline algorithms are implemented in [4]. Ours corresponds to adding Kron + MultiSkip networks. gc- means “goal-conditioned” and h- means “hierarchical”

Combined with the online RL results in the main paper, which involve compounded non-stationarity from both policy and value updates, these new results show that our approach: Is effective across different rates and sources of non-stationarity. Delays or prevents failure even under conditions where standard methods collapse. Our method not only tolerates significant non-stationarity, but also improves stability under a broad spectrum of dynamics.

Paper Formatting Concerns

Thanks a lot for bringing these points to our attention. We will incorporate all suggested corrections into the final version of the paper.

References

[1]. Obando-Ceron, J., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J. N., Dziugaite, G., Precup, D., and Castro, P. S.. Mixtures of experts unlock parameter scaling for deep rl. ICML’24

[2]. Lyle et al. "Understanding plasticity in neural networks." ICML’23

[3]. V Mnih et al. Human-level control through deep reinforcement learning, Nature’ 15

[4]. Park, Seohong, et al. Horizon Reduction Makes RL Scalable, arXiv’25.