Hyperspherical Normalization for Scalable Deep Reinforcement Learning

Dear reviewer fmAi,

Thank you for your thoughtful and constructive feedback. We address your concerns below and would be happy to clarify further.

Question 4.1 The scalability claims are not fully justified: (1) Only SAC is tested, (2) only width scaling is shown, (3) UTD scaling shows limited benefit.

(1) Algorithm: To evaluate generality beyond SAC, we ran SimbaV2 with DDPG on DMC-Hard and HBench-Hard (UTD = 2):

SimbaV2 performs comparably to Simba on DMC-Hard and significantly better on HBench-Hard, confirming its effectiveness beyond SAC.

(2) Depth scaling: We conducted additional experiments on DMC-Hard by varying the critic depth (1, 2, 4, 8 layers) using 5 random seeds.

Unlike SimbaV1, which degraded with depth, SimbaV2 improves consistently. This supports our view that effective regularization enables stable scaling in depth. Due to a limited time, HBench-Hard results are underway and will be included in the final manuscript.

(3) UTD scaling: While simple tasks (e.g., Cartpole-Balance) saturate quickly, more complex tasks such as HumanoidBench-Hard continue improving with higher UTD (see Fig. 6). Notably, prior work [1, 2] shows that high UTD often degrades performance unless combined with weight reinitialization. SimbaV2 maintains stable learning at high UTD without reinitialization, which we believe is a key contribution.

• [1] Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier, ICLR'23.
• [2] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, NeurIPS'24.

Question 4.2 It's unclear whether the reported metrics are necessary for stability and scaling. Many architectural choices in Section 4 are not motivated beyond empirical results.

The architectural decisions in SimbaV2 are based on training instabilities observed in SimbaV1 and supported by prior literature.

1. Feature Norm: TD learning introduces an implicit bias toward increasing feature norm [1], which can lead to overfitting and a reduction in feature rank. Both effects are closely linked to loss of plasticity during training [2, 3].
2. Parameter Norm, Gradient Norm, and ELR: As parameter norms grow, the ELR decreases, impeding gradient flow and leading to stagnation in learning [4]. This dynamic was observed in SimbaV1, particularly in the encoder.

These insights motivated the following designs in SimbaV2:

1. Hyperspherical normalization to control feature norm
2. Weight projection to constrain parameter norm.
3. Distributional critic with reward scaling to stabilize gradient norm.

Together, these components maintain a stable ELR across layers, preserve plasticity, and eliminate the need for weight reinitialization. We agree that this motivation was not clearly described and will revise the introduction to clarify these points.

• [1] DR3: Value-Based Deep RL Requires Explicit Regularization, ICLR'22.
• [2] Understanding and Preventing Capacity Loss in RL, ICLR'22.
• [3] Dissecting Deep RL with High Update Ratios: Combatting Value Divergence, RLC'24.
• [4] Normalization and effective learning rates in RL, NeurIPS'24.

Question 4.3 The use of a distributional critic is not defended.

As shown in this ablation, removing the distributional critic and reward scaling degrades performance and leads to a sharp decline in gradient norm, which in turn reduces the ELR. This supports its role in stabilizing training and preserving plasticity.

Question 4.4 What was the motivation behind Linear + Scaler?

Linear + Scaler serves two purposes: (i) weight projection controls parameter norm and ELR, and (ii) The learnable scalar amplifies important features.

Without the scalar, projection limits expressivity; without projection, training becomes unstable. Their combination is essential for balancing stability and flexibility.

Question 4.5 I learned very little from this paper, other than the fact that it works well.

Thank you for the candid feedback. Beyond empirical gains, our central insight is that stabilizing norm dynamics enables scalable, stable, and sample-efficient RL without relying on resets—a common workaround in the RL community. SimbaV2 offers a principled, architecture-level solution to this challenge.

We also believe these insights may transfer to other domains. For example, recent NLP work suggests long pretraining reduces model plasticity. SimbaV2’s techniques may inform the design of more robust architectures for large-scale training in other domains.

Dear Reviewer 4564,

Thank you for suggesting future research direction! We respond to each of your comments below and are happy to clarify further if needed.

Question 3.1 The claim that this leads to 'scalable' RL remains incorrect, as scaling limits are reached very quickly (See Fig. 5).

We appreciate your concern. While it is true that performance on DMC-Hard saturates earlier, this is largely due to task-specific ceilings rather than architectural constraints. Most tasks in DMC-Hard (except humanoid-run) reach near-optimal scores around 4.5 million parameters. In contrast, HBench-Hard continues to benefit from increased capacity up to 17.8 million parameters, demonstrating substantial headroom for scaling.

This level of parameter scalability is atypical in RL. Standard SAC or DDPG architectures often use around 2 million parameters, and larger models tend to degrade performance. In contrast, SimbaV2 scales consistently and robustly, without the need for reinitialization or tuning.

Similarly, in Fig. 6, compute scaling on DMC-Hard saturates around UTD = 4 due to task limitations, but HBench-Hard continues improving up to UTD = 8. This behavior highlights SimbaV2’s stable training dynamics under both model and compute scaling, which we believe justifies its claim to scalability within the current RL landscape.

Question 3.2 This paper only feels like a minor step up of SimbaV1. I would have liked to see experiments in a different set of environments, such as pixel-based control.

Why have the authors not attempted to do a performance analysis in at least a few pixel-based Mujoco tasks? How would the authors implement their techniques on CNN's ?

Thank you for this suggestion. We agree that applying SimbaV2 to pixel-based control is a promising direction. However, extending our techniques to convolutional architectures presents unique challenges. In CNNs, a shared kernel operates over overlapping spatial regions, making it nontrivial to enforce hyperspherical constraints on both features and weights, as we do with MLPs.

One possible approach is to project each C-dimensional fiber of the feature map and kernel onto a hypersphere, preserving the underlying normalization principle. However, this would require considerable architectural tuning, which we consider an important direction for future work.

That said, we believe the core ideas behind SimbaV2, controlling feature and parameter norms, and maintaining stable effective learning rates, can extend to vision-based RL. We look forward to exploring this as a future work.

Dear Reviewer ZJka,

Thank you for your constructive feedback and positive support! We respond to each of your points below and would be happy to clarify further if needed.

Q2.1: The necessity or effects of the design changes “Linear → Linear + Scaler” and “Residual Connection → LERP” are not supported with direct evidence.
Are there experiments for evaluating the effects of the design choices “the inverted bottleneck MLP”, “Residual Connection → LERP” and “Linear → Linear + Scaler” in this work?

Thank you for highlighting this. Below we provide both the rationale and supporting evidence for these design choices.

1. Linear → Linear + Scaler:
   This modification enables two key benefits: (i) weight projection ensures control over parameter norm and effective learning rate, and (ii) the learnable scalar allows selective emphasis of important features.

A standard linear layer without projection leads to unbounded parameter growth. On the other hand, projection without a scalar severely limits representational capacity. The Linear + Scaler combination is therefore necessary to balance norm control with expressivity.

2. Residual Connection → LERP:
   Since all features in SimbaV2 are normalized to lie on a hypersphere, standard residual connections are no longer applicable. The closest analog would be fixed-$\alpha$ interpolation (e.g., $\alpha$=0.5). We conducted an ablation to compare LERP vs. fixed-$\alpha$ residual:

LERP achieves higher performance in both benchmarks. We attribute this to two factors: its learnable mixing coefficient, which enables adaptive interpolation, and its initialization ($\alpha$=1/(L+1)), which biases early training toward identity mapping, enhancing stability.

Q2.2:
There are some other related papers not included:

• Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers. arXiv 2411.15370
• Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps. arXiv 2412.17113
• Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn. arXiv 2409.04792

Thank you for pointing out these relevant works. We appreciate the suggestions and will include them in the related work section. We have also added the following relevant works:

• Is High Variance Unavoidable in RL?., Bjorck et al., ICLR 2022
• Understanding, Predicting, and Better Resolving Q-Value Divergence in Offline RL., Yue et al., NeurIPS 2023
• Mixtures of Experts Unlock Parameter Scaling for Deep RL., Ceron et al., ICML 2024
• Dissecting Deep RL with High Update Ratios., Hussing et al., RLC 2024
• Don’t Flatten, Tokenize!., Sokar et al., ICLR 2025

Q2.3:
The performance of SimbaV2 is based on SAC, leaving its effects on PPO and DQN-variants unknown.

Thank you for highlighting this. While extending to PPO and DQN is a valuable direction, these algorithms present challenges: PPO is inherently on-policy and difficult to scale to high update-to-data (UTD) ratios, while DQN is restricted to discrete action spaces.

To assess the generality of SimbaV2 beyond SAC, we conducted additional experiments using DDPG, a widely adopted off-policy algorithm for continuous control:

In DMC-Hard, SimbaV2 performs competitively with Simba, both significantly outperforming the MLP baseline. In the more challenging HBench-Hard benchmark, SimbaV2 shows clear improvements over Simba, indicating enhanced stability and generalization beyond SAC.

Q2.4:
In Simba, the observation is centered and rescaled by running mean and std, and in SimbaV2, the observation is rescaled by the L2 norm. If there is a variant of SimbaV2 that only replaces the L2-norm rescaling with running std rescaling, how would it perform?

We appreciate the opportunity to clarify. There appears to be a misunderstanding: both Simba and SimbaV2 use running mean and standard deviation (RSNorm) for input normalization.

The key difference is architectural. SimbaV2 replaces LayerNorm with L2 normalization for internal features, not for input observations. This change targets internal stability and effective learning rates, while the input normalization strategy rem

Dear reviewer R7VW,

Thank you for your thoughtful and constructive feedback. We address your concerns in detail below and would be happy to clarify any remaining questions.

Question 1.1 It would be helpful if the authors could provide more theoretical justification or intuitive explanations for why previous methods struggle to scale while SimbaV2 does.

Why does hyperspherical normalization stabilize training and eliminate the need for weight reinitialization?

We appreciate this insightful question. Our use of hyperspherical normalization is motivated by empirical observations in SimbaV1, where we identified unstable growth in key quantities—including feature norm, parameter norm, gradient norm, and effective learning rate per layer (Fig. 4). These instabilities are known to impair generalization and reduce plasticity in RL.

Specifically:

1. Feature Norm: Prior work [1] shows that TD loss induces an implicit bias toward growing feature norms, which can cause overfitting and reduced feature rank. This is often driven by a few dominant dimensions, leading to loss of plasticity [2]. Techniques such as feature norm regularization [1] and hyperspherical normalization [3,4] were proposed to mitigate this.
2. Parameter Norm and Effective Learning Rate: As parameter norms grow, the effective learning rate (gradient norm divided by parameter norm) declines, which hampers gradient flow and learning dynamics [5,6]. In SimbaV1, we observed this specifically in the encoder, where the effective learning rate collapses over time (Fig. 4e). Simply increasing the global learning rate is not viable, as other layers (e.g., the predictor) may already be operating at high effective rates.

SimbaV2 addresses these challenges through the following design choices:

1. Feature Norm: Hyperspherical normalization to control feature norm growth.
2. Parameter Norm: Weight projection onto a hypersphere to control parameter norm.
3. Gradient Norm: Distributional critic with reward scaling to regulate gradient norm

Together, these mechanisms ensure stable learning dynamics (i.e., stable effective learning rate across layers) and sustained plasticity, thereby eliminating the need for weight reinitialization. We recognize that our original draft did not clearly explain these intuitions and will revise the introduction accordingly to highlight these design motivations.

• [1] DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization, Kumar et al, ICLR'22.
• [2] Understanding and Preventing Capacity Loss in Reinforcement Learning, Lyle et al, ICLR'22.
• [3] Is high variance unavoidable in rl? a case study in continuous control, Bjorck et al, ICLR'22.
• [4] Dissecting Deep RL with High Update Ratios: Combatting Value Divergence, Hussing et al, RLC'24.
• [5] Loss of plasticity in deep continual learning, Dohare et al, Nature'24.
• [6] Normalization and effective learning rates in reinforcement learning, Lyle et al, NeurIPS'24.

Question 1.2 Could the authors disclose the computational resources used for this project, including hardware specifications and training time?

All experiments were conducted using NVIDIA RTX 3090 GPUs and an AMD EPYC 7402 24-Core Processor.

For wall-clock time, a single run of SimbaV2 with UTD=2 typically takes 1.6 hours. This varies by environment: around 1.0 hour for simpler tasks like Cartpole (no collisions, minimal joints) and up to 2.5 hours for more complex tasks like Dog (many joints and intricate interactions).