Value-Based Deep RL Scales Predictably

Thank you for the feedback and a positive review of the paper. We are glad that you find our evidence clear and convincing. We answer your questions below. Please let us know if you find your concerns addressed, and if so we would be grateful if you would be willing to raise your score. We are happy to address any remaining concerns.

What is the wall-time efficiency of value-based algorithms in terms of scalability? For example, How long does the SAC algorithm take to complete a 5M environment-step training in DMC?

The wall-clock efficiency of these algorithms is highly dependent on the implementation of both the agent and simulator, as well as the computing infrastructure. For an NVIDIA A100 GPU, UTD=1 configuration, and DeepMind Control Suite of tasks (which present a CPU-based simulator), training for 5M environment steps should take around 12 hours. In our study, we model compute by making it proportional to the product of the model size, training length, UTD, and batch size (equation 4.2 of our manuscript) akin to how we measure FLOPs for language model training. As such, our compute metric is highly correlated to actual wall clock requirements. To avoid confusion, we will add an explanation to our compute metric definition where we discuss its relation to wall clock requirements, as well as add wall clock graphs of our tested algorithms to the appendix. Thank you for the suggestion!

Does it improve the memory usage when using the optimal hyperparameter

Our study uses a fixed memory size between all variants of algorithms (i.e. we use a fixed model and replay buffer sizes). As such, we left the study of memory/scaling interaction for future work. We hypothesize that the optimal or “good enough” hyperparameters indeed improve the memory usage of the algorithm, in a sense that it allows for training a successful agent with either less data or less parameters.

Beyond the changes described above, we add a number of improvements suggested by other reviewers (e.g., additional quantitative analysis of proposed scaling curves, estimation of scaling laws for different levels of J), which we invite you to inspect under the following link: https://sites.google.com/view/value-based-rl-scales/ and have released the code here: https://colab.research.google.com/drive/1BaqvAMb6svGojAuiOV8qFAUrZQwfPlDg?usp=sharing. We hope that these changes increase the reviewers' confidence in our work. If so, we kindly ask to consider updating the score of our work.

Thank you for the detailed review and feedback! We have added several new results & clarifications to address the concerns. Specifically, for the main points:

• We have performed the requested additional analysis to improve clarity here: https://sites.google.com/view/value-based-rl-scales/
• We eliminated inconsistencies in evaluation protocol and report the additional results on the same link
• To ensure transparency and reproducibility we have released the code: https://colab.research.google.com/drive/1BaqvAMb6svGojAuiOV8qFAUrZQwfPlDg?usp=sharing.

We think that these changes substantially strengthen the paper and are glad that we concur on the vision of understanding how to predictably scale value-based RL. However, we also think some of the comments might stem from a misunderstanding of certain parts of the paper, and we attempt to clarify these below. We will update the paper to clarify these. Please let us know if you find your concerns addressed and if so, we would be glad if you are willing to raise your score.

what J is

As specified on Line 139, J denotes the return of the policy normalized between 0 and 1000 (with normalization detailed in Table 1).

unclear whether the result holds for all values of J

It does: https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.mlu2zpb9cilb

hand-picked values of J, inconsistent number of observations
We standardized the number of observations in this plot to be 10 and standardized the values of J to be equally spaced in log space: https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.2u2k4w2tndmu.

It is unclear how accurate the result is.

We provide an additional extrapolation result where we use our model to predict the data/compute required to reach J (https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.erwyxjer0f42). The error is only 7.8% and 10.6% for extrapolating toward larger data and larger compute, respectively.

y-axis follows an unknown exponential scheme

Following prior work (Kaplan’20), y-axis follows a logarithmic scale. To improve readability, we also created a version that uses logarithm with base 2 (https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.8e53cf8zcw66).

unclear how the result was averaged (...), unclear whether the result holds over individual environments

We shared fits across several environments to increase the generality and robustness of the results (paragraph in line 391 and Appendices B & D). On IsaacGym, we also show that it is possible to obtain a good fit on a single task.

challenges associated with offline RL and high UTDs

Following DroQ, we use LayerNorm in our SAC implementation and BRO uses LayerNorm and resets by default. This allows higher utd. We further add a new experiment with high UTD of 64: https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.mo01yp5qtkz0. Here, we use the existing hyperparameter fits based on utd 0.25...8, extrapolating hyperparameters far beyond that, and we find that the data requirement fit is still accurate.

It is unclear how accurate the relationship is between UTD and best batch size/learning rate.

We provide quantitative correlations here https://sites.google.com/view/value-based-rl-scales/home?authuser=3#h.9t39ptle0tkt. We also provide quantitative evaluation of a baseline approach that uses hyperparameters optimal for UTD=2 to extrapolate to larger UTD.

It is unclear how the compute is affected by the unknown variable quantity B(σ). It is unclear how the best batch size or learning rate is determined over a collection of environments.

Best choice batch size is provided in Eq (4.6). The workflow used to obtain the value is provided in section 4.3. We use different hyperparameters per environment.

It is unclear how these results [for Eq 4.5] are generated.

The description the reviewer provides is correct. Please also refer to Section 4.3 for exact workflow.

How are empirical values [for Eq 4.5] averaged over a collection of environments?

This is described in l864. We normalize the data requirements by per-environment median.

Figures 3 and 1 (right) do not directly show UTD.

Please refer to the corresponding points in Figure 2.

The line of best fit is based on points sampled on another line of best fit (each pareto frontier), rather than empirical samples.

It is not possible to use empirical samples since the grid search includes only a small number of UTDs. Estimating scaling laws, however, enables us to optimize hyperparameters with higher precision than the grid search, similar to LLM literature (Kaplan’20, Dubey’24).

There are an inconsistent number of seeds used.

We used the maximum number of seeds that was feasible given our compute limitations. Different algorithms allow different number of seeds to be run in parallel.

Thank you for the review and the positive feedback. Please let us know if the response below addresses your concerns, and if there are any concerns remaining.

Although the evaluation criteria is fitting for the problem at hand, I would have liked to see some experiments on harder, pixel-based environments such as Atari. It would be interesting to see if these scaling laws hold with increased environment or network complexity

Our research vision is to understand the impact of various design choices on scaling of value-based RL algorithms. This includes a number of choices such as the choice of environments, networks, replay buffers, algorithms, etc, but unfortunately due to the limited compute budget we were left with the choice of only being able to study a few design choices only that could fit in the scope of the first paper along this direction. Therefore, we decided to first start from environments that are cheap to run (state-based) and considered different network architectures including small vanilla MLPs (used by SAC and PQL) and relatively big resnet-based models (BRO) in their specific domains.

We agree that expanding to pixel-based environments would further strengthen the generalizability claim. While this is costly to do, we are happy to add preliminary results with a subset of Atari environments in the camera-ready version.

In addition, in the rebuttal we have added a number of improvements suggested by other reviewers (e.g., additional quantitative analysis of proposed scaling curves, estimation of scaling laws for different levels of J), which we invite you to inspect under the following link: https://sites.google.com/view/value-based-rl-scales/ and have released the code here: https://colab.research.google.com/drive/1BaqvAMb6svGojAuiOV8qFAUrZQwfPlDg?usp=sharing. We hope that these changes increase the reviewers' confidence in our work.

Thank you for valuable feedback regarding our work. In accordance with your suggestions, we have made a number of changes to our manuscript. These include expanded discussion of limitations and experimental design. We discuss these in detail below:

(other parameters such as model size, weight decay, and target network update rates are also known to influence training stability and performance in deep RL

While studying a number of these factors is definitely along our vision of better understanding the behavior of scaling value-based RL algorithms along various axes, we were faced with a decision of only studying certain axes for the submission due to a limited compute budget. Therefore, we chose to initiate our study into this line of work with the most fundamental hyperparameters such as batch size, learning rate, and UTD values. We are now working to extend this line of work to study the impact of model size and target network update rates as the next piece of work along this research vision. We will also add the following discussion to limitations of this paper:

Whereas this study investigated the relationship between learning rates, batch sizes, and UTDs, there are a variety of other design choices and hyperparameters that could potentially impact the scaling profile of a given algorithm. In particular, previous works have shown that model size, weight decay, and target network update strategy can significantly affect the training dynamics between different UTD values.

It was surprising that the compute-data pareto frontiers (Figure 1, left) for OpenAI Gym seems to be much flatter than other environments. Do the authors have any intuition about this phenomenon?

Indeed, it seems that the effectiveness of UTD is benchmark-specific, where increased UTD leads to pronounced sample efficiency improvements in some environments, and substantially less pronounced in others. We believe that studying the underlying phenomena that lead to such differences is an exciting avenue for future research.

(...) The rationale for choosing SAC, BRO, and PQL is weak and not fully justified. SimBa [1] offers a more practical and performant alternative…

While in general, one can study scaling laws for any value-based RL algorithm, we were faced with the tough choice of only studying a few algorithms due to limited compute. We chose SAC, BRO, and PQL because prior work has shown them to be effective in the respective benchmarks (Gym, DMC, Isaac Gym) under different UTD configurations. We will add this justification to the paper.

Our framework is independent and can be applied to both BRO and SimBa, which are both methods focused on scaling the number of parameters. We observed consistent findings across a wide class of SAC-derived methods and therefore believe the results would also extend to SimBa. Due to compute constraints, we were unable to run SimBa experiments for rebuttal.

We also note that our goal is not to argue that our scaling laws give rise to the best possible value-based RL algorithm or that we are able to attain the best possible results in our experiments, but rather our point is to simply show that we can predict the behavior of value-based RL in larger-scale settings using small-scale experiments. We believe that showing this for a certain class of algorithms is still valuable as a starting step in this direction of research. We believe future state of the art methods will use our framework to improve performance, closing the loop.

Additionally, the paper does not compare against other well-established state-of-the-art methods such as MR.Q

Thank you for pointing us towards that work, which we now cite. We agree that studying the scaling capabilities of this general algorithm would be particularly interesting, however we were not able to include MrQ in our submission as it was arXived after the ICML deadline.

However, more works on scaling model size [1] and UTD ratio [2, 3] in RL could be included:

Thank you for pointing these missing references, we added all of them to our manuscript.

Beyond the changes described above, we add a number of improvements suggested by other reviewers (e.g., additional quantitative analysis of proposed scaling curves, estimation of scaling laws for different levels of J), which we invite you to inspect under the following link: https://sites.google.com/view/value-based-rl-scales/ and have released the code here: https://colab.research.google.com/drive/1BaqvAMb6svGojAuiOV8qFAUrZQwfPlDg?usp=sharing. We hope that these changes increase the reviewers' confidence in our work. If so, we kindly ask to consider updating the score of our work.