Compute-Optimal Scaling for Value-Based Deep RL

Thank you for your positive and constructive feedback. To address your concerns, we are including several new experiments and discussion as requested, which we believe further strengthen our submission. In particular, we are adding new results on different architectures beyond BRO, n-step TD, and evaluations of scaling on Atari games, a domain quite distinct from robotics. Please let us know if these responses address your concerns, and if so, we would be grateful if you are willing to raise your score.

BRO architecture?

Our decision to focus on the BRO architecture was motivated by findings in prior work [1,2], which show that naively scaling critic models often leads to degraded performance. BRO incorporates architectural and regularization components (i.e. BroNet) that have been shown to enable stable critic scaling with consistent performance gains. We agree that regularization can influence the behavior of TD error, and chose BRO to study TD overfitting and scaling in a setting where scaling is feasible. Our findings and the fitted scaling laws should therefore be interpreted as conditional on an architecture that supports scalable TD learning. This type of conditional framing is consistent with prior work in model scaling, such as scaling laws for dense transformers [3], mixture-of-experts models [4], or state-space architectures [5], where scaling behavior is reported for architectures known to scale well. To clarify this for readers, we now emphasize in the main text that our scaling laws are specific to the BRO architecture, and the parameters of our fitted curves depend on regularization and architectural design.

Robotic tasks?

We chose to focus primarily on robotic tasks because our goal is to investigate whether model scaling in value-based RL can yield predictable improvements, similar to trends observed in supervised learning. This is a non-trivial question as prior work [1,2,6,7] has highlighted that naively increasing model size or UTD often leads to unstable or degraded performance in RL. The considered robotic tasks are commonly used in recent RL model scaling studies [2,6], enabling direct comparison and contextualization of our results.

That said, we agree that evaluating scaling behavior in other domains, such as discrete action environments with value-iteration-style algorithms, is an important direction. To this end, we are currently running a subset of Atari-100K experiments with DQN, and preliminary results shown in the table below confirm that model scaling can yield monotonic improvements in performance in discrete-action tasks as well and we will expand this with the TD overfitting analysis in the camera-ready version. Furthermore, we have expanded the limitations section to explicitly state that our current findings may not generalize across all RL settings (e.g. goal-conditioned, sparse reward, or pixel-based tasks), and require further validation.

Continuous action space?

We thank the reviewer for this suggestion. We agree that continuous action tasks introduce additional approximation challenges, particularly when estimating the TD target via action sampling or optimization (as opposed to exact maximization in discrete settings). This does make theoretical analysis of TD error more complex. However, our main motivation for focusing on robotic control tasks is to study TD overfitting in high-dimensional, realistic settings where RL is frequently deployed, e.g. in applications such as sim-to-real. To this end, we include experiments on challenging, high-dimensional environments, such as the simulated Unitree robot from HumanoidBench, to ground our analysis in realistic control settings. We agree complementing this with discrete action experiments would offer a useful theoretical contrast, and we are working toward that in parallel (see our response above on Atari-100K).

n-step TD?

We thank the reviewer for this insightful suggestion. We agree that, theoretically, increasing $n$ in multi-step TD should reduce the susceptibility to TD-overfitting - as $n \rightarrow \infty$, the learning target approaches a Monte-Carlo return estimate, effectively turning the problem into supervised regression on full returns rather than bootstrapped targets. In this sense, 1-step TD represents the most bootstrapped and thus most overfitting-prone setting. However, in the off-policy setting, multi-step TD introduces additional bias, especially when learning from older transitions that reflect outdated or suboptimal policies. In practice, this tradeoff between bias and variance can confound the expected reduction in TD-overfitting as well. To this end, our preliminary experiments with the BRO architecture suggested that n-step TD improves performance with small models, where variance reduction helps stability. However, performance at large n degrades with larger models, likely due to increased off-policy bias. We are currently running more comprehensive experiments to validate these observations and plan to include these results in the final version.

[1] Bjorck, Nils, et al. "Towards deeper deep reinforcement learning with spectral normalization." NeurIPS 2021

[2] Nauman, Michal, et al "Bigger, Regularized, Optimistic." NeurIPS 2024

[3] Kaplan, Jared, et al. “Scaling Laws for Neural Language Models”. arXiv 2020

[4] Krajewski, Jakub, et al. "Scaling laws for fine-grained mixture of experts." arXiv 2024

[5] Gu, Albert and Dao Tri. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." COLM 2024

[6] Lee, Hojoon, et al. "Simba: Simplicity bias for scaling up parameters in deep reinforcement learning." ICLR 2025

[7] D'Oro, Pierluca, et al. "Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier." ICLR 2023

Thank you for the feedback and a positive assessment of our work. To address your concerns, we performed new experiments on alternative network designs for the critic and studied scaling of learning rates for training as reported below. Further, we would like to thank you for additional comments on self-supervised learning and double descent, which look at our work from new and interesting perspectives – we will make sure to add these into our discussion section. In this paper, we will focus on strengthening the existing claims, however, these alternative perspectives might lead to wider impact in future work.

Different network designs

We ran 5 Atari-100K tasks with DQN with a 2-layer MLP at UTD=1, and for each $J$ estimated the data efficiency for the best batch size. Our preliminary results show that data efficiency improves with model size on average.

We agree that regularization can influence the behavior of TD error, and chose BRO to study TD overfitting and scaling in a setting that has shown consistent performance gain with critic scaling. Our findings and the fitted scaling laws should therefore be interpreted as conditional on an architecture that supports scalable TD learning. This type of conditional framing is consistent with prior work in model scaling, such as scaling laws for dense transformers [1], mixture-of-experts models [2], or state-space architectures [3], where scaling behavior is reported for architectures known to scale well. To clarify this for readers, we now emphasize in the main text that our scaling laws are specific to the BRO architecture, and the parameters of our fitted curves depend on regularization and architectural design.

Learning rate scaling

We have now run an additional analysis of learning rate dependencies. We find that (i) best learning rate decreases with increasing model size, correlation: -0.75, (ii) best learning rate decreases with increasing UTD, correlation: -0.46, (iii) best learning rate increases with increasing batch size, correlation: 0.42. These results are consistent with existing literature McCandlish’18, Yang’22, Rybkin’25. We are running additional experiments to evaluate this hypothesis and will report back before the end of the discussion period.

Below, we show simple log-linear fits of the best learning rate. We empirically find that the best learning scales less than linearly with the batch size.

h1-crawl-v0: $R^2$ = 0.3417
Relative error: 43.5702%
$\text{lr}^* ~ 4.4827\text{e}-4 \cdot (N/2.3\text{e}6)^{-0.3112} \cdot \sigma^{-0.1273} \cdot (B/512)^{0.3709}$

h1-stand-v0: $R^2$ = 0.5702
Relative error: 32.2800%
$\text{lr}^* ~ 2.4727\text{e}-4 \cdot (N/2.3\text{e}6)^{-0.2472} \cdot \sigma^{-0.2392} \cdot (B/512)^{0.2701}$

Performance-optimal vs. compute-optimal scaling

Our framework addresses this issue of finding the “reasonable batch size and optimal UTD ratio to attain the best final performance” as described in Problem 3.1, part 2 and Appendix A.3. Specifically, our framework can be applied in both ways: (i) given a certain performance, minimize the budget required to reach it, and (ii) given a certain budget, maximize performance that can be achieved. Both of these applications can be derived from the functional form of the scaling law that we study. Practitioners can solve for either question as they find necessary, we do not intend to imply that one question is more important than the other.

self-supervised learning methods like BYOL or MoCo

In this paper, we focus on a studying of SOTA value-based RL methods, which we believe is crucial for future work on large-scale RL. It is an interesting point that self-supervised learning can have some similarities with the analysis proposed in our paper. We would like to note that in value-based RL, an additional source of potential instability arises from the fact the actor is updated continuously, so the values change even faster and due to more factors than in self-supervised learning. It is likely that the value-based RL setting is more challenging. However, self-supervised learning is more mature and allows for larger-scale experiments than value-based RL, and so is an exciting direction of future work.

Double descent phenomenon

This is a great question! We have not observed a double descent phenomenon in the range of model sizes we evaluated. We would like to note that two complications appear in value-based RL that might impact the double descent presence. First, the notion of overfitting is different in value-based RL since overfitting can occur due to noise in the targets, as discussed in our paper. Second, the notion of network capacity is different, since the network must not only fit the targets at the end of training, but also all targets as they change during training, requiring more capacity. Because of this, further work is needed to analyse whether double descent is present in value-based RL.

Offline RL applicability

We expect that some of our findings transfer directly to offline RL as the phenomenon of TD-overfitting should be similarly present. However, due to increased distribution shift between the data and the policy, offline RL has other complicating factors such as the need to tune a ‘conservatism’ coefficient or a BC regularization loss. Addressing this is an exciting direction of future work. We will add this to the discussion section of the paper.

[1] Kaplan, Jared, et al. “Scaling Laws for Neural Language Models”. arXiv 2020

[2] Krajewski, Jakub, et al. "Scaling laws for fine-grained mixture of experts." arXiv 2024

[3] Gu, Albert and Dao Tri. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." COLM 2024

[4] Rybkin, Oleh et al. “Value-Based Deep RL Scales Predictably.”

[5] Yang, Greg. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

[6] McCandlish, Sam. “An Empirical Model of Large-Batch Training.”

Thank you for the comprehensive and thoughtful response. We are glad to hear that our paper ‘provides a clear empirical investigation’ that ‘that could be useful […] for practitioners’. To address your points about improving the empirical evidence presented in the paper, we provide additional experiments to address the feedback. Due to large compute requirements and a delay in obtaining the required compute resources to rerun hyperparameter sweeps used to analyze scaling behaviors, some of our experiments are still running. We will report back and include them when they finish, within the author-reviewer discussion period.

We address the remainder of the concerns below.

Learning rate is completely ignored.

We have now run an additional analysis of learning rate dependencies. We find that (i) best learning rate decreases with increasing model size, correlation: -0.75, (ii) best learning rate decreases with increasing UTD, correlation: -0.46, (iii) best learning rate increases with increasing batch size, correlation: 0.42. With simple log-linear fits, we obtain a relative error of 37.5%. These results are consistent with existing literature McCandlish’18, Yang’22, Rybkin’25. We are running additional experiments to evaluate this hypothesis and will report back before the end of the discussion period.

h1-crawl-v0: $R^2$ = 0.3417
Relative error: 43.5702%
$\text{lr}^* ~ 4.4827\text{e}-4 \cdot (N/2.3\text{e}6)^{-0.3112} \cdot \sigma^{-0.1273} \cdot (B/512)^{0.3709}$

h1-stand-v0: $R^2$ = 0.5702
Relative error: 32.2800%
$\text{lr}^* ~ 2.4727\text{e}-4 \cdot (N/2.3\text{e}6)^{-0.2472} \cdot \sigma^{-0.2392} \cdot (B/512)^{0.2701}$

Despite the high relative error, we observe that data efficiency is similar within an interval of “reasonable” learning rates, which includes our fixed value; see the “high relative error” section below. By using the optimal learning rate fit in our results, we can improve the method’s performance. We expect this method to follow the same data efficiency law as Eq (6.1), but have better coefficients $\alpha_J, \beta_J$, because Eq (6.1) is justified with asymptotic case analysis and is consistent with results from Rybkin’25. We are currently running experiments to fit this improved data efficiency law and will report them once they finish.

A complete N-dimensional analysis of learning rate was not possible in this rebuttal period due to computational requirements: 4 learning rate values x 4 batch size values x 4 model size values x 4 utd values yields 256 required GPUs per environment. However, our initial analysis provides insight into how learning rate behaves and can be used by future studies to propose more effective scaling methods.

other domains or algorithms.

In addition to the two domains and 17 tasks in the original submission, we have now evaluated scaling laws on DQN in Atari. This is consistent with prior work on architectures [1,2] and RL scaling laws [3] which both showed findings generally applicable to many algorithms and environments. Interestingly, we observe that on Atari, batch size needs to be decreased with model size, which can be estimated from small-scale experiments using our workflow. We are investigating whether this is due to statistical overfitting or TD-overfitting and will report and include these results when they finish. Future work will focus on validating scaling laws on practical large-scale environments.

We ran 5 Atari-100K tasks at UTD=1, and for each $J$ estimated the data efficiency for the best batch size. Our preliminary results show that data efficiency improves with model size on average.

Unclear evidence of improved performance.

In Figure 4, we show the compute-optimal settings for UTD and model size for each data budget. As we show in Appendix A.2, this scaling method is also data-optimal for each compute budget. For each compute budget $\mathcal C$, we run the configuration with UTD $\sigma^\*(\mathcal C)$, model size $N^\*(\mathcal C)$, and batch size $\tilde B(\sigma, N)$. We provide three baselines demonstrating the superior performance of compute-optimal scaling: for each compute budget, we can additionally run (i) UTD-only scaling at compute budget $\mathcal C$ for a given model size, (ii) N-only scaling at compute budget $\mathcal C$ for a given UTD, (iii) the compute-optimal UTD and model size and constant batch size (specifically, the fitted batch size from the lowest compute budget).

In the table, we record the ratio between the data efficiency in each baseline to the compute-optimal data efficiency, aggregated over environments and compute budgets. We show that compute-optimal scaling with the fitted batch size outperforms the baselines.

We further note that we expect the improvement from our approach to be more prominent as the gap between small-scale settings that we can readily experiment with and large-scale settings that we wish to find good hyperparameters for diverge from each other. For instance, tuning hyperparameters on a 1e6 parameter model and transferring to a 5e6 parameter model might be possible since the original hyperparameters are already in a good neighborhood; but transferring to a 2e9 parameter model (bare minimum for a language application) will likely work poorly unless a proper scaling law procedure is applied and hyperparameters are adjusted. As the only work establishing joint scaling laws for batch size, UTD, and compute, we believe future practitioners will rely on our results in deciding how to tune hyperparameters at large scale.

high relative error in the fits (50%)

Indeed, we observe that our batch size fits have a moderate confidence interval of up to 50%. We observe this is because batch size in many cases has a wide range of values which all have near-optimal performance. Note that utilizing any of these batch values performs similarly, which means a practitioner can indeed use any value within this interval to obtain good performance and a moderate confidence interval is justified. In the table below, we group runs by UTD and model size, and bin suboptimal batch sizes. Then, we consider the data efficiency ratio between suboptimal and optimal runs, and average over UTDs and model sizes. We find that batch sizes within an interval around the best batch size $B^*$ perform reasonably, and performance degrades significantly with larger intervals. This is consistent with prior work on LLMs [4], and explains the relative error in the fit. In practice, this means that our fits, despite a large relative error, yield good results when using them to train models, which is anyways our ultimate goal.

Similarly, we observe that there is a range of reasonable learning rates:

In practice, we ran our main experiments at a constant learning rate of 3e-4. Running an additional sweep over learning rates of [1e-4, 2e-4, 6e-4] showed that only 2e-4 had superior performance in some cases. Since we are within the [lr*, 1.5 lr*] interval, the overall effect on performance is minimal.

We believe the submission is now much stronger after addressing your feedback. Please let us know if there is anything else in the submission that you think we should address for an improved score.

[1] Nauman, Michal, et al "Bigger, Regularized, Optimistic." NeurIPS 2024

[2] Lee, Hojoon, et al. "Simba: Simplicity bias for scaling up parameters in deep reinforcement learning." ICLR 2025

[3] Rybkin, Oleh et al. “Value-Based Deep RL Scales Predictably.”

[4] Grattafiori, Aaron et al. “The Llama 3 Herd of Models.”

[5] McCandlish, Sam. “An Empirical Model of Large-Batch Training.”

[6] Yang, Greg. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Thank you for the thorough and engaging feedback, and for a positive assessment of our work. We believe that the alternate perspectives and references that you discuss are quite relevant, and we will discuss them in the updated version of the paper. We discuss your concerns below and we have also performed additional experiments to address some of the concerns.

interference between updates

Thanks for pointing out this paper, we will cite it and discuss it in the updated version. Briefly, Bengio’20 discusses a phenomenon where TD learning leads to low interference due to a lack of temporal coherence in the value network. This is consistent with our analysis: a lack of temporal coherence suggests a poor target distribution, which in turn yields overfitting. A more detailed study of this phenomenon is warranted. However, a complicating factor is that measuring value changes can be challenging in continuous environments, and absolute value changes might not correspond to meaningful updates in the parameter space. We will perform proof-of-concept experiments on this and include them in a later version.

target network update rate

Intuitively, target update rate should be important for the TD-overfitting phenomenon. In our preliminary experiments, we found that when holding other hyperparameters constant, TD-overfitting increases when setting the target update rate too high. That said, we saw that the effect of the target network update rate was overshadowed by setting a different learning rate and running training at a different UTD value & batch size. In other words, our initial experiments indicated that instead of changing the target network update rate we could find an alternate configuration of other hyperparameters that gave the same performance, and the target update rate was less necessary to account for.

Given a limited amount of computational resources, we had to make a conscious decision early on in the project regarding which hyperparameters to focus on and given the above preliminary findings, we decided to skip the target update rate. A full study of such effects is an exciting direction of future work, and we will rerun some of the preliminary experiments into the appendix of the final version of the paper. We do agree that there might be some nuanced effects of the target network update rate on TD-overfitting, which would be good to study in future.

Do you expect similar findings to hold for different learning algorithms

We ran 5 Atari-100K tasks with DQN at UTD=1, and for each $J$ estimated the data efficiency for the best batch size. Our preliminary results show that data efficiency improves with model size on average.

We believe Atari is more susceptible to overfitting due to the deterministic nature of the environments. However, in general, since DQN also relies on TD-learning, we expect similar trends to apply. In contrast, PPO, when run with standard hyperparameter settings, relies on policy gradients more than TD-learning. Therefore, we expect that PPO scaling laws will be more similar to supervised learning scaling laws and simpler. Studying whether PPO is also affected by the TD-overfitting phenomenon we describe is an interesting direction of future work, particularly for applications using policy gradient methods, such as LLMs.

The notation is not clearly explained

We use $\alpha, \beta$ for the power coefficients and $a, b$ for multiplicative coefficients. The subscripts specify the equation that the terms appear in, e.g. $a_B$ in $B(., .)$, $a_J$ in $D_J(. ,.)$, $a_C$ in $C^*(.)$, etc.

Fig. 3 shows curves fit to only 4 data points each

We will provide heatmaps in the appendix. Note that the fit in Eq (5.1) has 4 variables, and is fit to 16 points. All 16 points are shown in Fig 3. We have considered different functional forms and selected this one as it had the best empirical performance and intuitive justification as discussed on l223. Specifically, with large $\sigma$, it will dominate the denominator, requiring a small batch size, consistent with Rybkin’25. With small $\sigma$, the model size term will dominate, consistent with Kaplan’20.

In Fig. 4: How are the contours generated?

We plot predictions from the fit (6.1). Because the fit is two-dimensional (it depends on $\sigma$ and N), we are able to produce a continuum of predictions, which is necessary for proper extrapolation.

Is the actor network also scaled up

In line with prior work [1], we found that scaling critic-only leads to best performance.

Please let us know if there is anything else in the submission that you think we should improve or clarify for an improved score.

[1] Nauman, Michal, et al "Bigger, Regularized, Optimistic." NeurIPS 2024

[2] Rybkin, Oleh et al. “Value-Based Deep RL Scales Predictably.”

[3] Kaplan, Jared, et al. “Scaling Laws for Neural Language Models”. arXiv 2020