Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning

We are grateful for your thoughtful review and for recognizing the importance of our research. Your suggestions encouraged us to significantly strengthen both the theoretical and empirical components of the paper.

Weakness

W1: There are almost no formal guarantees nor formal explainations why GraMa works. I know that this might be difficult, but could you come up with any kind of formal guarantees or explainations for GraMa? This would make the paper even stronger.

Thanks for your valuable feedback. We appreciate your recognition of our paper’s strengths while acknowledging, as you noted, the inherent challenge of developing comprehensive and realistic theoretical guarantees for complex deep RL systems.

As the first study to comprehensively analyze gradient magnitudes for neural maintenance in deep RL, our work primarily establishes a strong empirical foundation across multiple modern architectural settings in deep RL, with a few theoretical justifications, though we fully agree that these could be better highlighted and expanded.

Based on our continued theoretical explorations since submission, we have identified several promising future theoretical directions for our work, and we will include these discussions in the updated manuscript: 1) the relationship of resetting parameters with small gradient norms and suboptimal saddle point escape, which is a common challenge in deep RL optimization landscapes [1, 2], 2) the effect of selecting parameters with minimal gradient magnitudes for resetting on disruption to learned features while maximizing learning capacity. The developments in these theoretical perspectives can further help explain why GraMa consistently outperforms activation-based alternatives across diverse architectures and tasks, but remains particularly challenging due to the complex learning dynamics in deep RL systems. Our work also opens several promising avenues for future theoretical exploration including the connection to second-order optimization, and convergence guarantees in non-stationary environments.

In our revision, we will better highlight these theoretical insights with our paper's strong empirical focus and theoretical justifications. We believe our pioneering work not only establishes a practical method for neural maintenance but also provides valuable theoretical foundations for this emerging area of deep RL.

[1]: Ge, Rong, et al. "Escaping from saddle points—online stochastic gradient for tensor decomposition." COLT , 2015.

[2]: Mishkin, Dmytro, et al. "All you need is a good init." ICLR, 2015.

Minor Weakness

W1: I suggest removing the double title "Complex Architecture Based Policy" which is, anyhow, misleading in my view. Also revise the caption on the y-axis on the top subfigure. What is the unit here? That is not entirely clear. Also make sure that the axis labelling is readable. Currently, it is too tiny.

Thanks for the suggestion. We incorporated recommended changes in the revision, where we removed the duplicate title, revised the caption to clarify the metric (the y-axis represents the score normalized by the peak ReGraMa performance, with the maximum value being 1.0), and enlarged axis labels for better readability.

W2: The notation in Eq. 1 looks a bit overly complex. Consider simplifying. But if it is common and taken from the corresponding papers, you can also leave it as it is.

Thanks for raising this point! To ensure consistency with established literature [1], we retained the original notation, as it facilitates direct comparison with prior work.

[1]: Sokar et al., The Dormant Neuron Phenomenon in Deep Reinforcement Learning, ICML 2023

W3: Fig. 3 is impressive but a bit more description would help the understanding. What are the light-blue and light-red areas?

Thanks for highlighting this! We have enhanced the visualizations with clearer labels: light-blue areas now explicitly mark neurons with low learning capacity and low expressivity—the primary candidates for reset, while light-red indicates neurons with high learning capacity but low expressivity. We also added new panels that explicitly show the performance impact of resetting each of the four neuron types, thereby highlighting their distinct functional roles (detailed results are provided in Response W3 to Reviewer 1 due to space limitation).

W4: What is the unit of measurement on the y-axis in the subfigure at the center of Fig. 5? Also, in Fig. 5 the axis labelling and the headings of the center and right-most subfigures are unreadable small.

We have enlarged all fonts and added explicit labeling showing that the y-axis represents percentage improvement. For intuitive interpretation, we normalized each ReGraMa score relative to the score of ReDo (ReDo’s score = 1.0), so the y-axis directly shows ReGraMa’s relative improvement over ReDo in percentage.

W5: The message in the text (Lines 215-220) does not really correspond to the visual information in Fig. 6. The figure suggest that during training, already inactive neurons become even less active...

In the revised manuscript, we now state: “The figure indicates that neurons that are already inactive at the start of training become progressively less active as training proceeds.”

W6: Sometimes the figures could be displayed closer to the places in the text where they are discussed, e.g., Fig. 5.

Great suggestion! We reorganized the layout so all figures appear adjacent to the corresponding discussion, eliminating the need for flipping pages and improving overall readability.

W7: In Fig.9 the axis labelling is too small. Also, the meaning of 1x, 2x and so on on the x-axis is not clear. How many layers do these networks actually have?

In the revised Figure 9, we enlarged all axis labels and revised the x-axis labeling with a clear caption explaining: model sizes begin with two BRO-net blocks (as per the standard setting), and larger model scales are constructed by adding additional BRO-net blocks in sequence.

Question

Q1:

Please refer to W1.

Thanks again for reviewing our work and constructive feedback! We hope our responses address your concerns. We are happy to provide additional clarification if needed.

We would like to sincerely thank the reviewer once again for the recognition and support of our work, and for the time and effort dedicated to reviewing our manuscript!

We will carefully incorporate your valuable feedback into the revision, which has greatly helped us improve the quality, clarity, and overall contribution of our work.

We would like to express our deep gratitude for your significant support for this article. Your detailed suggestions have enabled us to further improve our work. We hope that our response can address all your concerns.

Weakness

W1: Why does the gradient magnitude imply “learning capacity” and the activation magnitude imply “expressivity”? This is not well discussed in the paper.

Thanks for the question. We have expanded our background section in the revised version, to provide a clearer explanation and foundations of learning capacity and expressivity.

• Recent work [1] defines learning capacity as an agent’s ability to efficiently adapt its parameters to the current data distribution. Gradient magnitude directly quantifies this property; it reflects the effect of parameter updates prescribed by back-propagation upon receiving learning signals with new data. Hence, smaller magnitudes indicate lower learning capacity.
• In [2,3], expressivity refers to the model's representational ability for the inputs. Empirical studies show that this correlates with the proportion of neurons whose activations exceed a threshold. Accordingly, subsequent work adopts mean activation magnitude as a practical proxy for expressivity.

[1]: Liu et al., Neuroplastic Expansion in Deep Reinforcement Learning, ICLR 2025

[2]: Sokar et al., The Dormant Neuron Phenomenon in Deep Reinforcement Learning, ICML 2023

[3]: Ma et al., Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning, ICML 2025

W2: Figure 3: This seems like circular reasoning—you define "learning capacity" as the gradient magnitude and then evaluate GraMa, which is based entirely on that same metric. Doesn’t this place ReDo at an unfair disadvantage? That said, the results do show that the two methods reset different types of neurons.

We use GraMa for analysis because ReDo tends to misidentify dormant neurons in complex architectures. Specifically, ReDo systematically flags a large fraction of healthy neurons as dormant while missing those that empirically harm performance. This misalignment is clearly visible in the reset-preference visualizations in Figures 1 and 3. As such, activation-based scores lose their statistical power in these cases. To facilitate direct visual comparison, we have added ReDo curves to the main experimental plots in the revised manuscript.

W3: Suppose a neuron consistently receives a high-magnitude gradient (greater than \tau) but in alternating directions (e.g., +c, -c, and so on). In such a case, can we truly say that the neuron is “learning'' or that it has high “learning capacity''?

Our definition of learning capacity focuses on the magnitude of the gradient to reflect the agent’s "willingness" to update parameters. Even if the update direction oscillates, the neuron remains actively engaged in learning. Therefore, we consider large magnitudes, regardless of sign, as indicative of high capacity. In practice, measuring this is more tractable and stable than evaluating directional correctness on a per-neuron basis.

W4: (Line 158) “Less expressive at the moment” — As I understand it, [Sokar et al., 2023] show that dormant neurons tend to remain dormant over time. Does this observation not apply to the BroNet architecture discussed in this context?

Indeed! In the revised manuscript, we clarify that architectures with normalization (e.g., BRO-net) are already known to partially mitigate dormancy. In these cases, ReDo’s activation-based metric can occasionally “revive” dormant neurons, which is visible as minor oscillations in ReDo’s dormant-neuron curves. This transient rebound becomes more pronounced in deeper or more complex networks.

W5: (a,b) I recommend adding the Resnet SAC architecture in the Appendix as well (since all the others are present) ...

We have added the Resnet SAC architecture in Appendix B of the revised manuscript and reorganized this point.

W5: (c) The analysis in Section 3 discusses how ReDo fails to identify dormant neurons in the settings considered (e.g., non-ReLU activations). To validate this claim, I suggest an experiment comparing the performance of SAC with ReLU and TanH activations, both with and without ReDo—i.e., SAC-ReLU, SAC-ReLU-ReDo, SAC-TanH, and SAC-TanH-ReDo. If ReDo improves performance in the ReLU setting but not with TanH, it would provide empirical support for the claim.

We followed your suggestion and replicated the experimental setup from ReDo’s paper on the MuJoCo Humanoid task (1M steps, Table 6). We normalized all results against SAC-ReLU for clearer comparison. The table below shows the performance ratios (mean ± std across 3 runs).

The results in the table below match your conjecture. ReDo does improve the performance of the agent compared to vanilla ResNet-SAC under ReLU, but degrades it under TanH.

Table: Advantages over SAC-ReLU (Average of 3 runs):

W6: Figure 7(a) —Are ReDo and ReGraMa equivalent in this setting (according to Theorem 1)? Shouldn't we then expect them to perform similarly? (ReGraMa gets a higher reward)

Thank you for pointing this out. We apologize for the lack of clarity. The results in Figure 7 are based on BRO-net, not a standard MLP+ReLU architecture. Thus, GraMa and ReDo are no longer equivalent, and the superior robustness of ReGraMa is reflected in its higher reward. We clarify this distinction in the new version.

W7: I also recommend that the authors include plots of the \tau-dormant ratio for some of the reported results.

We have added $\tau$-dormant ratio plots for ReDo in the revised manuscript to better visualize how it compares with GraMa under different conditions.

W8: Are “Low learning Capacity neuron ratio” (Figure 1) and GraMa the same?

Yes, they are equivalent. We will explicitly clarify this in the revised version.

W9: (Typos)

We have thoroughly revised the manuscript to fix typos, improve clarity, and correct inconsistent notation.

Questions

Q1: Can you address the questions regarding "learning capacity" and "expressivity"?

Please refer to our response to W1 above.

Q2: Could you also plot the \tau-dormant neuron ratio?

Yes, we will include these plots in the revised manuscript.

Q3: Could you include the results mentioned in Weakness 5(c) to support the "Analysis" section?

Yes, please refer to our response to W5(c) and the accompanying experimental table.

Thanks again for your valuable feedback. We appreciate your thoughtful suggestions and hope our clarifications are helpful. Please let us know if anything remains unclear.

We would like to sincerely thank the reviewer once again for their helpful and constructive comments and for the time and effort dedicated to reviewing our work.

We will carefully incorporate your valuable feedback into the revision, which has greatly helped us improve the quality, clarity, and overall contribution of our work.

We thank the reviewer for the insightful suggestions and useful feedback on our work. Please find our responses to each of the concerns below.

Weaknesses

W1: The distinction between GraMa-detected "inactive" neurons and ReDo-detected "dormant" neurons is not rigorously analyzed. What is the distributional difference between neurons flagged as "inactive" by GraMa and "dormant" by ReDo? What is their overlap rate?

Thanks for raising this point! We have conducted additional experiments by performing a more thorough distributional analysis to examine the differences between neurons identified by each method. We evaluated this on the BRO-net architecture on DMC Humanoid Walk, and will include the full results in the new version.

1. We measured the overlap between neurons flagged by ReGrama and ReDo. The average overlap rate is only 31.56% (average of 3 runs), confirming that they identify fundamentally different neuron populations.
2. We tracked the proportion of neurons reset by ReDo and ReGraMa throughout training. As shown in Table 1, ReDo performs aggressive resets early in training when activation patterns are naturally unstable (potentially leading to falsely identifying neurons). In contrast, ReGraMa shows a more progressive pattern (which resets more neurons as training evolves) that better aligns with the gradual loss of learning capacity (indicative of a more conservative and adaptive strategy).

Table 1. Reset neuron ratios throughout training (average of 3 runs):

1. We analyzed the layer-wise distribution for each method. Table 2 shows nearly opposite layer-wise preferences: ReDo predominantly resets neurons in input/output layers, whereas ReGraMa focuses more on the middle layers (where representational learning primarily occurs).

Table 2. Layer-wise reset neuron ratios at step 500k (average of 3 runs):

W2: Although Theorem 1 proves … there is no theoretical analysis in complex architectures with different activation functions.

Thanks for bringing up this point. While Theorem 1 establishes equivalence for MLPs with ReLU activations, we acknowledge that comprehensive theoretical analysis for complex architectures (like ResNet) with arbitrary activation functions remains a significant open challenge in the field. Despite this, our approach has strong theoretical intuition: gradient magnitudes better adapt to any differentiable architecture’s computational graph, which can reflect neuron contribution regardless of the activation function. Our empirical results also strongly support this intuition, showing consistent performance improvements across diverse architectures and activation functions. We hope our work opens a new venue for future theoretical advances that can formally characterize dormancy detection across arbitrary architectures.

W3: For complex network architectures and different activation functions, there are too few experiments for empirical verification.

We have expanded our evaluation to address this concern and further substantiate our conclusions, which highlight the efficiency of ReGraMa.

• We compare ReGraMa with four related methods based on RNN architectures, where ReGraMa consistently outperformed strong baselines including CBP, ReDo and S&P (refer to Q4)
• We further compare ReGraMa with the four mentioned recent methods, i.e., ReBorn, CBP, S&P and ReDo, based on BRO-net, where ReGraMa demonstrates superior performance (refer to Q5).
• We conducted experiments on non-ReLU activation setting by comparing with the four additional baselines, where ReGraMa consistently maintains its performance advantage (refer to Q5).

Questions

Q1:

Please refer to our response to W1 above.

Q2: I want to know the impact of resetting these four types of neurons on performance. Such analysis is very important to prove the effectiveness of GraMa.

To analyze which neurons contribute to ReGraMa’s performance gains, we added a study isolating the impact of resetting each neuron type from Fig. 3. The table below shows the following key findings:

1. Resetting neurons with high learning capacity and low expressivity harms performance, where these neurons are frequently misidentified by ReDo as dormant due to their low activity. Furthermore, resetting those with high learning capacity and high expressivity causes the most substantial performance degradation.
2. Resetting neurons with low learning capacity and high expressivity yields marginal improvements, while resetting low learning capacity + low expressivity neurons provides the largest performance boost.

This analysis demonstrates that ReGraMa’s effectiveness stems from its ability to discriminate between neurons based on learning capacity.

Table. Performance impact (percentage change relative to the vanilla BRO-net policy, avg of 3 runs):

Q3: Why ReGraMa reduces GraMa significantly better than ReDo with exactly the same experimental settings in Fig. 5?

There appears to be a misinterpretation of Fig5 (right), where the first figure in Fig5 (right) shows that ReGraMa (orange) and ReDo (pink) show nearly overlapping GraMa metrics throughout training with only minimal differences (approximately 0.001), and the 2nd figure in Fig5(right) shows that both methods achieve comparable accuracy curves on CIFAR-100. The slight variations are attributed to randomness within the model rather than algorithmic distinctions, which is expected and consistent with our theoretical analysis.

On the other hand, Fig. 5 (left) demonstrates a different message, which is about the significant computational efficiency advantage of GraMa over $\tau$-dormant neuron calculation.

Please kindly let us know if we understand this question correctly.

Q4: I checked the code you provided, ... Does ReGraMa work with RNNs, a common architecture in critic networks?

We further extended ReGraMa to GRU-based RNN architectures based on ReBorn’s official implementation [1]. ReGraMa achieves excellent effect as ReBorn, consistently outperforming most baselines.

Table: Performance on RNN-based agents (avg. of 3 runs)

[1]: The dormant neuron phenomenon in multi-agent reinforcement learning value factorization.

Q5: Other plasticity-aware baselines (e.g., CBP, ReBorn) should be included beyond ReDo.

We have extended the experiment on BRO-net and TanH activation-based methods for comparison based on 4 DMC tasks.

The results based on the BRO-net are summarized in Table a. ReGraMa consistently outperforms other methods and highlights its robustness in complex architecture. On the other hand, ReBorn, which considers both over-active and dormant neurons, shows moderate improvements and struggles with certain tasks (Dog walk), while S&P and CBP show slight improvements.

Table a. Normalized score relative to baseline based on BRO-net (average of 3 runs):

In the non-ReLU setting, ReGraMa and CBP demonstrate strong and robust performance, where our method achieves more stable performance across different tasks.

Table b. Normalized score relative to baseline based on TanH based ResNet-SAC (average of 3 runs):

These comprehensive results across different neural plasticity methods further demonstrate ReGraMa's robustness across architectures, activation functions, and task domains.

Thanks again for reviewing our work and constructive feedback! We hope our responses address your concerns. We are happy to provide further clarification if needed.

Dear Reviewer AyZs,

Thanks once again for your time in reviewing our work and for the constructive suggestions!

As the author–reviewer discussion period concludes in less than 24 hours, we would like to briefly follow up to confirm that our subsequent responses resolve your concerns. Any further comments would be greatly appreciated. Thanks!

Best, The Authors

Thank you for the constructive review and for noting that the additional experiments addressed all your concerns! If the concerns are now resolved, we would be grateful if you could update the score accordingly, as we believe the improved clarity and validation meet the bar for acceptance, in line with your own final assessment.

We will carefully incorporate your valuable feedback into the revision, which has greatly helped us improve the quality, clarity, and overall contribution of our work.

Thank you again for your time and effort.

Thank you for your thoughtful review and detailed feedback! We carefully considered your suggestions and conducted additional experiments and analyses to address your concerns. Below we respond to each point.

Weaknesses

W1: The paper's central idea has been developed before as "gradient dormancy" measure for variable-strength Shrink & Perturb resets in ACE. Therefore, I cannot consider this a novel contribution of the author's paper.

Thank you for highlighting this relevant prior work! After a detailed comparison and new empirical analysis, we believe our contribution remains novel and significant. We have added a discussion of similarities and differences between ACE and our work to the revised manuscript.

Key Differences with ACE [1] in focus, scope, and mechanism and our new insights:

• ACE primarily proposes a novel causality-aware entropy term to improve exploration, with a secondary and empirical gradient dormancy component to prevent overfitting that guides Shrink & Perturb resets (which is implemented in a “global” way, and please find our new experimental analysis below for further explaining the difference in this part).
• In contrast, our work provides a comprehensive framework that provides novel insights for understanding neuronal learning capacity in deep RL. We provide the first rigorous analysis of why popular activation-based metrics fail with modern deep RL architectures and identify new phenomena. We show how gradient information resolves these issues and the widely adopted ReDo measure can be restored effectively if properly guided by gradient magnitude through our principled approach.
• We provide detailed analysis and new perspectives that reinterpret the failure as stemming from ReDo’s statistical objective, not its mechanism, and examine how modern DRL architectural components affect this previously overlooked phenomenon.

Additional empirical comparison:

Following the reviewer’s suggestion for a more thorough evaluation, we have expanded our comparisons to include more recent strong baselines across diverse architectures (see W2) and non-ReLU activations (see W3); and investigated the differences in learning behavior (see W5) as requested. Uncover the neuron-level gradient behavior during training from the time and model perspective(see response to W1 of Reviewer AyZs).

We compare ReGraMa and ACE's reset method regarding the generalization on diverse architectures. The consistent improvements demonstrated by ReGraMa across diverse architectures validate our theoretical insights and demonstrate their practical significance.

Table: Advantages over base policy at 3M step on Walker Walk (average of 3 runs):

In summary, our work provides a rigorous, novel, and in-depth foundation, analysis, and comprehensive evaluation of neuronal learning capacity in deep RL, addressing key challenges previously unexplored in the literature.

[1]: Ji, Tianying, et al. "Ace: Off-policy actor-critic with causality-aware entropy regularization." arXiv preprint arXiv:2402.14528 (2024).

W2: The baselines are weak... there should be more baselines (not all are needed): Full resets, Plasticity Injection, and resets according to ACE.

We have added the related strong baselines including full resets, Plasticity Injection and Shrink & Perturb on 4 DMC tasks (6M steps). All methods follow the official implementations. Results show that ReGraMa improves learning performance across all 4 tasks. The radical approach of full resets magnified performance uncertainty, as evidenced by a high standard deviation across different seeds. Plasticity Injection performs the weakest, likely because abrupt architectural changes can interfere with learning in complex agents. Shrink & Perturb reset shows stable but relatively minor gains in 3 tasks.

Table. Performance improvement over baseline (Average of 3 runs):

W3: When using TanH, which is most common for state-based continuous control, the benefits of ReGraMa are minimal over a vanilla baseline (Figure 12).

We replaced MuJoCo with a more challenging task (Walker Walk) based on SAC. The results below (normalized to vanilla SAC) show clear benefits from ReGraMa, outperforming both ReDo and the vanilla policy:

Table. Advantages over vanilla policy (Average of 3 runs):

Besides, we followed ReDo's setup in Appendix B and adopted the sparse-SAC to avoid the noise caused by parameter redundancy. The results show that our method significantly improves the policy performance.

Table. Advantages over vanilla policy (Average of 3 runs):

W4: The authors claim that "Recent studies [Nauman et al., 2024b, Lee et al., 2025a] … improve representation capability on complex visual tasks". BRO is not a pixel RL agent; this statement must be clarified.

We would like to clarify that our claim is based on Table 8 of the official BRO-net paper (camera-ready, OpenReview NeurIPS’24), which evaluates BRO-net on image-based tasks. We will highlight that we use the same encoder setup (DrQv2-style convolutional encoder) and BRO’s official PyTorch wrapper.

W5: The introduction mentions catastrophic forgetting as motivation to discard reset approaches, but then proceeds to not evaluate catastrophic forgetting ... Therefore, I believe reset approache to be a valid baseline for ReGraMa.

We use DrQ as the backbone and compare the performance on the long-term training task Cheetah Run (8M steps). It can be seen that full resets may cause instability and oscillations. ReGraMa balances the restoration of network activity with forgetting by precisely resetting individual neurons.

Table. DrQ on Cheetah Run (Average of 3 seed):

We also added PPO as the on-policy baseline to compare the above methods. ReDo reset has a relatively weak positive effect on the performance of neurons. Full resets caused performance instability, while ReGraMa significantly enhance learning efficiency.

Table. PPO Performance on HalfCheetah (Average of 3 runs):

We would like to emphasize that "mitigate catastrophic forgetting" is mainly used to introduce the contributions of related work, not to describe the approach proposed in this paper.

W6: Sometimes the writing is rather verbose. For example, the introduction is rather lengthy.

We’ve streamlined the introduction and moved the detailed related work to its own dedicated section.

Questions

Q1: How dependent is ReGrama on different optimizers?

We uniformly use the Adam optimizer for fair comparison.

We have tested ReGraMa with different optimizers on Walker Walk. The optimizers include Adaptive Moment Estimation (Adam), Stochastic Gradient Descent (SGD), and Root Mean Square Propagation (RMSprop). Results show that ReGraMa might be more suitable for optimizers with adaptive learning rate, though future work could explore optimizer-reset interactions further.

Table. Optimizer Dependence (Average of 3 seeds):

Q2: How does ReGraMa interact with parameter regularization, such as weight decay or L2 init [3]?

The effectiveness of the combination ReGraMa with weight decay (follow the optimal setting in [1]) and L2 init ($\lambda=1e-2$) is tested using DrQv2. Results show that integrating either method with ReGraMa can further enhance learning efficiency. L2 init may be more compatible with ReGraMa.

Table. ReGraMa with Regularization (Average of 3 runs):

[1]: Lyle, Clare, et al. "Disentangling the causes of plasticity loss in neural networks." arXiv (2024).

Thank you again for your constructive feedback. We hope these additional results address your concerns. Please let us know if further clarification is needed.

Thank you for the extensive rebuttal, particularly the additional experiments. The new baselines address my request for broader empirical comparison, and I will raise my score by one to reflect this.

However, my concerns about the paper’s novelty remain. In my view:

our work provides a comprehensive framework that provides novel insights for understanding neuronal learning capacity in deep RL

ReGraMa combines ACE’s gradient-dormancy criterion with the ReDo reset, replacing Shrink & Perturb. While this performs better, it does not amount to the “comprehensive framework” claimed.

We provide the first rigorous analysis of why popular activation-based metrics fail with modern deep RL architectures and identify new phenomena

The failure of activation-based metrics in continuous control was already demonstrated by ACE. Your results on BRO-net provide a minor extension of ACE's insights for a particular network architecture. The authors of ACE have already shown that gradient dormancy works on more modern architecture (see Appendix J, results with DrQ-v2). To me, the main new phenomenon established by your paper is the permanent loss of learning capacity over time, which is minor.

We provide detailed analysis and new perspectives that reinterpret the failure as stemming from ReDo’s statistical objective

The stated insight that ReDo’s issues stem from its statistical objective is the same as ACE’s original motivation for introducing gradient dormancy (("However, the dormant phenomenon defined in these works is not evident in state-based RL", p.4), so I do not see a genuinely new conceptual contribution.

In short, the paper is technically solid and now empirically stronger, but I still find the conceptual advance incremental. Thus, after the score increase noted above, my recommendation remains below the acceptance threshold.

Thank you again for the clarifications and for the new experiments.

I still see the foundational contributions as coming from ACE. Namely, 1) revealing the failure of activation-based metrics in common Deep-RL settings, 2) proposing a gradient-based alternative, and 3) showing that gradient-based approaches can transfer across architectures.

That said, your paper extends ACE's findings: you

• broaden the evaluation to a wider range of architectures
• analyze the temporal dynamics of (gradient) dormant neurons
• apply a different (but already established) reset formulation that yields practical gains
• show that gradient dormancy subsumes activation dormancy in a stylized ReLU-MLP setting.

To reflect these additional contributions and your new experiments, I have raised my score by one.

As our views on the relative novelty of the core insight and contributions still differ, I suggest we agree to disagree and let the final decision rest with the chair.

We greatly appreciate the reviewer's recognition that our work is 'technically solid and now empirically stronger' according to our additional experiments and analysis during the subsequent responses, which led to raising your score. Thanks once again for your time and input which has helped improve our manuscript.