1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

We thank Reviewer 4F8D for the helpful feedback, questions, and suggestions.

1. How do the training costs of the proposed algorithm compare with those of the baselines? Can authors show it in a table?

Sure! Because the baseline algorithms use standard-sized networks (less than 4 layers), our scaled approach naturally incurs higher raw wall-clock time per environment step (Table 1a). However, for practical deployment the more meaningful metric is the total wall-clock time required to attain a given level of performance. To evaluate this, we compare our method with the strongest baseline, SAC. As shown in Table 1b, our approach surpasses SAC performance in less wall-clock time in 7 out of 10 environments.

2. Can the authors elaborate empirically or conceptually on why performance “jumps” occur at specific depths or why agents exhibit different behaviors at certain depths (as shown in Figure 3) in certain environments?

To elaborate conceptually why performance “jumps” occur at specific depths, we draw parallels to the broader deep learning literature of the well-documented phenomenon of emergent abilities, where a sudden increase in performance arises at a critical threshold of model scale. Wei et al. 2022 [1] first proposed that large language models exhibit such emergent behavior, with performance transitioning sharply from near-random at a certain threshold points in model size. Although cross-entropy loss improves smoothly with scale, the model only begins to complete tasks once a critical threshold of capability is reached. In our experiments, we observe similar patterns in the Humanoid agent: as the depth increases from 4 to 8 to 16 layers, its ability to locomote improves gradually. However, only upon reaching a critical level of coordination—when it can walk stably without falling—does the agent achieve the goal and remain upright. This manifests as a sharp, discontinuous jump in performance. Wei et al. also remark that emergent abilities often involve qualitatively novel behaviors, such as chain-of-thought reasoning in LLMs, that are absent in smaller models. We observe a parallel in the Humanoid U-Maze environment: only at a very large model scale (i.e., depth 256) does the agent achieve the complex and coordinated movement necessary to successfully vault over the wall. At smaller depths, the agent simply appears to “flounder”, its movements not yet sophisticated enough to complete the maneuver. This behavior manifests as a qualitatively distinct, “emergent” ability and is reflected in the observed performance jump.

3. There is an error in the citation of the appendix at line 173, which shows as "？？".

Thank you for pointing that out. This figure comparing our approach with baselines is included as Figure 10 of the Appendix. The issue was due to a typo that resulted in an undefined reference — we have now fixed this in our local version.

4. The paper does not mention the training and inference costs of the proposed algorithm. The training and inference costs associated with a network depth of 1000 layers may be relatively high.

Thanks for this suggestion. In terms of considerations on training and inference costs, we report the wall-clock time of our approach as we scale depth (Table 2a, Table 2b). A comparison of the cost of wall-clock time vs. baselines is discussed in the first paragraph of this response above, and a comparison of wall-clock time for depth vs. width scaling can be found in Table 3a and Table 3b. We likewise also report the metrics of FLOPs and memory (Table 4, Table 5).

5. For example, why does the performance undergo a sudden change when the depth reaches a certain critical point? This, to some extent, limits the article's insight.

We address this in the second paragraph of this response above.

[TABLE 1a] Wall-clock Time Comparison of Our Method vs. Baselines

[TABLE 1b] CRL Surpasses SAC Performance in Less Wall-clock Time in 7 of 10 Environments

*N/A denotes SAC outperforms our approach, so CRL never reaches SAC performance. These account for 2 of the 3 environments where CRL fails to surpass SAC in less wall-clock time.

[Table 2a] Wallclock time of Depth 4, 8, 16, 32, and 64 across all 10 environments

[Table 2b] Wallclock time up to Depth 1024 in the Humanoid U-Maze environment

[TABLE 3a] Wall-clock Time Comparison of Depth vs. Width Scaling

[TABLE 3b] Scaling Depth Surpasses Performance of Scaling Width in Less Wall-clock time

[TABLE 4] Comparison of Width vs. Depth Scaling: FLOPS and Memory

FLOPs comparison of width vs. depth scaling

[TABLE 4] Comparison of Width vs. Depth Scaling: FLOPs

[TABLE 5] Comparison of Width vs. Depth Scaling: Memory

[1] Wei, J., et al., NeurIPS 2022, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models https://arxiv.org/abs/2201.11903

We thank Reviewer kTrp for the helpful feedback, questions, and suggestions.

1. How is training time affected as you scale depth?

We report the wall-clock metrics of our approach as we scale depth in Table 1. We also conduct a comparison to baselines: since the baselines use standard sized networks, naturally our scaled approach incurs higher raw wall-clock time per environment step (Table 2a). For practical deployment, however, the more meaningful metric is often wall-clock time to attain a given level of performance. In Table 2b, we compare against the strongest baseline, SAC, and show that in 7 out of 10 environments our approach surpasses SAC performance in less wall-clock time.

2. Limited generality of the claims : my main concern with this work is that the idea of scaling depth seem to apply to a very specific algorithm, i.e. contrastive RL, and in very specific settings, i.e. online RL.

To address this concern, we report a new experimental result: we find that our approach successfully scales depth on a different algorithm (GCBC) in a different setting (offline RL). In Table 3a, we find that GCBC successfully scales on the antmaze-medium-stitch task in OGBench [1] when using our architecture of layer normalization, residual connections, and swish activations. In Table 3b, we show that using our architecture is critical to enabling scaling in GCBC, as scaling the standard architecture degrades performance. These results suggest that our approach to depth scaling may be adapted to other algorithms, as well as the offline setting. Future work could further investigate what algorithms and architectural components best enable depth scaling in offline RL.

3. Given that simply scaling depth in the actor seems beneficial, would scaling depth be useful for BC and GCBC in Offline RL settings?

Thanks for this suggestion! For GCBC, we have discussed the results in the response above. We also attempt to scale depth for standard BC in OGBench [1]. However, in this case we find negative results (Table 4).

4. I understand the authors provide a study on CMD-1 in the Appendix, which uses the MRN network for quasimetric learning. It would be useful to also study QRL [1], which instead uses IQE for quasimetric learning

We attempt to scale depth for QRL in both the online and offline settings, using the original QRL codebase [2] and OGBench [1] respectively. However, in both settings, we find that while QRL is able to train stably with deep networks, deep networks don’t improve performance beyond that of the shallow networks. Table 5a and Table 5b show these results.

[Table 1a] Wallclock time of Depth 4, 8, 16, 32, and 64 across all 10 environments

[Table 1b] Wallclock time up to Depth 1024 in the Humanoid U-Maze environment

[TABLE 2a] Wall-clock Time Comparison of Our Method vs. Baselines

[TABLE 2b] CRL Surpasses SAC Performance in Less Wall-clock Time in 7 of 10 Environments

*N/A denotes SAC outperforms our approach, so CRL never reaches SAC performance. These account for 2 of the 3 environments where CRL fails to surpass SAC in less wall-clock time.

[TABLE 3a] Depth Scaling on GCBC (Antmaze-medium-stitch, OGBench)

[TABLE 3b] Our Architecture is Critical: Depth Scaling on GCBC Fails with Standard Architecture

[Table 4] Depth Scaling on BC (Antmaze-giant-navigate on expert SAC policy, OGBench)

[Table 5a] Depth Scaling on Online QRL (FetchPush task on original QRL codebase)

[Table 5b] Depth Scaling on Offline QRL (Pointmaze-giant-stitch, OGBench)

[1] Park, S., et al., ICLR 2025, OGBench: Benchmarking Offline Goal-Conditioned RL, https://arxiv.org/abs/2410.20092.
[2] Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning

We thank Reviewer 5GqS for the helpful feedback, questions, and suggestions.

1. The experiments are limited to one type of architecture. Whether the same trend would occur in other well-designed architectures in RL [1,2,3] is unsure.

To address this concern, we have run a new experiment: we find that the same trend occurs when we apply our approach to another well-designed architecture for scaling RL, Simba-v2 [1]. The key innovation in Simba-v2 is replacing layer norm with hyperspherical normalization, which projects network weights onto the unit-norm hypersphere after each gradient update. Table 1 shows that indeed the same depth scaling trends holds when we add the hyperspherical normalization to our architecture, and, in fact, actually improves the sample efficiency of depth scaling. This suggests that our method is able to integrate new architectural innovations in the emerging RL scaling literature.

2. On a similar note, no sufficient analysis was made on the design choices of the architecture, other than the residual connection.

As suggested by the reviewer, we report new ablation experiments for the other design choices of architecture: namely, layer norm and swish activation. We show that both of these design choices are critical for effective scaling:

• In Table 2a, we show that removing layer normalization significantly hampers scalability.
• Additionally, Table 2b shows that scaling with ReLU activations performs significantly worse than Swish activations.

These results, along with Figure 5 of the manuscript, show that all of our architectural components—residual connections, layer norm, and swish activations—are jointly essential to unlocking the full performance of depth scaling.

3. The connection between depth and exploration is not well explained. The results presented in figure 13 could also be due to improved learning capability of the collector.

To clarify, by exploration, we specifically mean the breadth or coverage of states visited. Shallow networks, due to their limited learning capacity, tend to explore only states near their initializations. While deep networks, due to their greater learning capacity, learn effective long-horizon behaviors, resulting empirically in much broader state coverage. Figure 13’s shallow collector experiments illustrate that improved learning capacity alone is insufficient when data diversity or coverage is poor. Conversely, Figure 13’s deep collector experiments show that good data coverage alone, without adequate learning capacity, also falls short. Thus, rather than viewing learning capacity and exploration as diametrically opposed, they mutually reinforce each other: stronger learning capacity drives more extensive exploration, and rich data coverage is essential to fully realize the potential of stronger learning capacity. Both aspects jointly contribute to improved performance.

4. Due to the model's sequential nature, a depth-scaled model is likely to take more time to generate output than a width-scaled model with same number of parameters. The claim that depth scaling is more compute efficient than width scaling (line 225:229) should thus we accompanied by a comparison on wall-clock time.

In Table 3a, we report the average wall-clock time for scaled width and depth, and find that width and depth empirically scale quite similarly in terms of wall-clock time. For practical deployment, the relevant metric is often wall-clock time to reach a given level of performance. Table 3b shows that scaling depth surpasses the performance of scaling width in less wall-clock time in all three environments.

5. The fact that their architecture does not scale in offline GCRL weakens their point.

Although we found negative results for scaling CRL in the offline setting, we now report new experiments demonstrating that our approach scales effectively to another widely used offline algorithm, GCBC. In Table 4a, we find that GCBC successfully scales with depth on the antmaze-medium-stitch task in OGBench [2] when using our architecture of layer normalization, residual connections, and swish activations. In Table 4b, we show that using our architecture is critical to enabling this scaling in GCBC, as scaling the standard architecture degrades performance. These results suggest that our approach may be adapted to scaling the offline RL setting. Future work could further investigate what algorithms and architectural components best enable depth scaling in offline RL.

[TABLE 1] Hyperspherical Normalization (Simba-v2) Improves Sample Efficiency of Depth Scaling

Steps to reach ≥ 200 success

Steps to reach ≥ 400 success

Steps to reach ≥ 600 success

[TABLE 2a] Layer Norm Ablation

[TABLE 2b] Swish vs. ReLU Ablation

[TABLE 3a] Wall-clock Time Comparison of Depth vs. Width Scaling

[TABLE 3b] Scaling Depth Surpasses Performance of Scaling Width in Less Wall-clock time

[TABLE 4a] Successful Depth Scaling on GCBC (Antmaze-medium-stitch, OGBench)

[TABLE 4b] Our Architecture is Critical: Depth Scaling on GCBC Fails with Standard Architecture

[1] Lee, H., et al., ICML 2025. Hyperspherical Normalization for Scalable Deep Reinforcement Learning. http://arxiv.org/abs/2502.15280.
[2] Park, S., et al., ICLR 2025, OGBench: Benchmarking Offline Goal-Conditioned RL, https://arxiv.org/abs/2410.20092

We thank Reviewer gFHA for the helpful feedback, questions, and suggestions.

Question 1: Can you provide any intuitive explanation to why the scaling benefits appear specific to CRL?

Sure! The key intuitive reason for scaling benefits specific to CRL lies in how it redefines the core learning challenge of RL. Traditional RL faces several well-known scalability challenges: regression to noisy and recursively bootstrapped Q-value targets, gradient instability, plasticity loss, amplifying overestimation biases, etc. In contrast, CRL fundamentally shifts the “heavy lifting” of the RL learning problem from recursively estimating fluctuating Q-values to addressing a simple classification problem: is the sampled future state likely to be along the same trajectory or along a different trajectory as my current state? This transforms the unstable RL problem into a supervised learning classification task based on cross-entropy loss—a setup known to scale exceptionally well in deep learning (and recently also in RL, see Farebrother et al., 2024 [4]). Once CRL scalably learns good contrastive representations through this supervised objective, the remaining learning burden on the actor becomes straightforward: at any given state, simply greedily choose the action that yields the representation most similar to the goal state.

Question 2: Could you provide detailed computational cost analysis comparing depth vs. width scaling? What are the FLOPS, memory, and wall-clock time trade-offs? How does this inform practical deployment decisions?

In Table 1a, we show that width and depth empirically scale similarly in terms of wall-clock time. For practical deployment, the relevant metric is wall-clock time to reach a given performance. Table 1b shows that scaling depth surpasses the performance of scaling width in less wall-clock time in all three environments. In terms of FLOPs and memory, width scales quadratically while depth scales linearly, making depth more efficient in these metrics as well. Thus, for practical deployment we recommend scaling depth in almost all cases.

Question 3: In Figure 1, the performance of these approaches increases with depth, and then seems to decrease when depth is increased further. What is the reason for such behavior? Do you have any thoughts on finding the critical depth for particular tasks?

Even with techniques like residual connections and layer norm, training very deep networks is known to be tricky—exploding gradients, representation staleness, plasticity loss, etc. This is likely why many RL scaling efforts show results on width scaling but not depth. In our experiments, for tasks where degradation does occur, performance has already scaled to near-optimal. Thus, we hypothesize that additional depth yields diminishing returns while sometimes increasing the risk of instability. In these cases, to answer the question on identifying critical depth:

• Start with 32 layers: Across all 10 environments, we consistently observe stable and improving performance up to depth 32.
• Scale deeper if needed: For more complex tasks, especially those involving locomotion or high-dimensional control (e.g., Humanoid environments), we observe stable scaling up to 1000 layers.
• Be conservative with manipulation tasks: These tasks tend to be more sensitive to over-scaling and performance degradation at higher depths.

Question 4: In line 173, the figure seems to be missing?

This Figure is in the Appendix (Figure 10). Thanks for catching the undefined reference.

Weakness 1: It seems that the scaling benefits appear specific to CRL. The traditional TD methods (SAC, TD3) show no improvement or even show degradation with depth.

This is a known result from prior work that traditional TD methods do not scale well with depth [5, 2, 3]. As such, rather than viewing this as a limitation, we see it as a point of strength/novelty of our work: our results demonstrate that the benefits of scaling depth can indeed be unlocked in RL. Previous efforts to scale RL have largely focused on network width [5,2], with depth scaling either limited (e.g., up to 8 residual blocks in [3]) or harmful [5, 2]. Our approach unlocks the ability to scale depth, yielding performance improvements greater than width alone.

Weakness 2: The preliminary experiments show no benefits in offline settings, limiting applicability.

Although we found negative results for scaling CRL in the offline setting, we now report new experiments demonstrating that our approach scales effectively to another widely used offline algorithm, GCBC. In Table 2a, we find that GCBC successfully scales with depth on the antmaze-medium-stitch task in OGBench [1] when using our architecture of layer normalization, residual connections, and swish activations. In Table 2b, we show that using our architecture is critical to enabling scaling in GCBC, as scaling the standard architecture degrades performance. These results suggest that our approach may be adapted to scaling the offline RL setting. Future work could further investigate what algorithms and architectural components best enable depth scaling in offline RL.

Weakness 3: The experiments are specific to unsupervised RL setting. It is unclear if the benefits may transfer to reward-rich (dense) or imitation learning settings.

CRL is an unsupervised RL algorithm; as such, we focus on this setting as the primary scope of our paper. One potentially interesting direction for future research: although CRL itself is unsupervised, its self-supervised contrastive representations can be learned on any set of collected trajectories, including those of a reward-based RL algorithm. Prior work has explored integrating such representations via, for example, auxiliary losses or planning [6]. Our analysis of visualized Q-functions and learned representations give confidence that increasingly sophisticated representations can be learned through depth scaling. We encourage future work to investigate whether these scaling benefits can thus be integrated into reward-based RL.

Weakness 4: The paper would benefit from detailed analysis of computational overhead compared to baselines.

Because the baselines use standard-size networks, our scaled approach naturally incurs higher wall-clock time per environment step at a raw computation level (Table 3a). However, as with the width-vs-depth analysis, the more meaningful metric is the total wall-clock time required to reach a given performance level. To evaluate this, we compare our method with the strongest baseline, SAC. As shown in Table 3b, our approach surpasses SAC performance in less wall-clock time in 7 out of 10 environments.

Weakness 5: The approach requires substantial computational resources that may limit reproducibility.

In fact, all of our experiments, including those with 1000 layer networks, can be run on a single 80GB A100!

[TABLE 1a] Wall-clock Time Comparison of Depth vs. Width Scaling

[TABLE 1b] Scaling Depth Surpasses Performance of Scaling Width in Less Wall-clock time

[TABLE 2a] Successful Depth Scaling on GCBC (Antmaze-medium-stitch, OGBench)

[TABLE 2b] Our Architecture is Critical: Depth Scaling on GCBC Fails with Standard Architecture

[TABLE 3a] Wall-clock Time Comparison of Our Method vs. Baselines

[TABLE 3b] CRL Surpasses SAC Performance in Less Wall-clock Time in 7 of 10 Environments

[1] Park, S., et al., ICLR 2025, OGBench: Benchmarking Offline Goal-Conditioned RL, https://arxiv.org/abs/2410.20092
[2] Lee, H., et al., ICLR 2025, SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning. http://arxiv.org/abs/2410.09754
[3] Lee, H., et al., ICML 2025. Hyperspherical Normalization for Scalable Deep Reinforcement Learning. http://arxiv.org/abs/2502.15280
[4] Farebrother, J., et al., ICML 2024, Stop Regressing: Training Value Functions via Classification for Scalable Deep RL. http://arxiv.org/abs/2403.03950
[5] Nauman, M., et al., NeurIPS 2024. Bigger, Regularized, Optimistic: Scaling for compute and sample-efficient continuous control. http://arxiv.org/abs/2405.16158
[6] Laskin, M., et al. ICML 2020, CURL: Contrastive Unsupervised Representations for Reinforcement Learning. https://proceedings.mlr.press/v119/laskin20a.html

I appreciate the clarifications and the detailed computation analysis and scaling based analysis. The authors' response has addressed all my concerns, and I will maintain my already high judgement score for the paper.