We thank the reviewer for suggesting valuable additional ablations and experiments. We added the new results to the updated paper, and we provide our detailed answers below.

1. Additional REDQ and DroQ Ablations

The evaluation looks promising but some experiments are missing to yield a well-rounded study. For example, CrossQ updates the Adam momentum and critic width compared to REDQ and DroQ. How do REDQ and DroQ compare to CrossQ when we set their (1) Adam momentum to 0.5, (2) critic width to 2048, (3) momentum to 0.5 and width to 2048? To reduce compute requirements, this could be studied for e.g., 5 seeds on Humanoid + HumanoidStandup + 1 other task

These are all good suggestions. We performed each of these experiments on 5 seeds for all environments, and added them to a dedicated section in the Appendix (Figures 14 & 15). We find that these changes do not improve the baselines’ performance, and therefore, we retain the better performing versions of the baselines in the main body of the paper.

2. Generality of the Proposed Method

The concept should be sufficiently general to combine with other baseline algorithms than SAC, so evaluation across multiple algorithms would further broaden applicability [i.e.are the results specific to SAC variations?]

This is an interesting observation. We based CrossQ on SAC because our baselines REDQ and DroQ are also based on SAC. However, our concepts are directly applicable to other off-policy TD algorithms. For demonstration, we added a TD3 [1] baseline, as well as a version of CrossQ based on TD3 to Figure 5. This combination shows the same effects in terms of increased sample efficiency, confirming that our proposed design choices regarding batch normalization and target networks can be applied more broadly and are not specific to SAC and the max-entropy RL setting

[1] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 2018.

3. Additional Environments

Generally, a more diverse set of environments would further strengthen the evaluation – this could include other domains (DeepMind Control Suite / MetaWorld / …) or visual control tasks [i.e.are the results specific to OpenAI Gym domains?]

To align our statements with the experiments, we have narrowed the claims within the paper by specifying more exactly that our results are restricted to continuous-control tasks with state observations. In our original submission, we included the same benchmark suite as the REDQ and DroQ papers, and even reported experiments on 6 environments rather than 4 as in the previous papers. In our updated submission, we double our number by including 6 additional environments on the DeepMind Control Suite, revealing the same trend in the improved sample efficiency provided by CrossQ. However, note that we had a limited time during the rebuttal to tune CrossQ as well as the baselines on those environments, and better results may be achievable till the camera-ready submission deadline.

We note that a thorough evaluation of CrossQ on vision-based tasks is in general beyond the scope of the present paper and will constitute a separate publication. In the current paper, we focus on establishing CrossQ as a state-of-the-art algorithm for systems with state-based observations, closely following the setting of REDQ and DroQ. We foresee an extension of CrossQ to vision-based tasks as the next step for future work.

4. Additional CrossQ Ablations

Have you tried removing the double Q functions on top of removing the target networks?

We have added this experiment to the ablation study in Fig. 9, however, we find that removing the double Q function in fact hurts the performance.

Have you tried keeping the target network and normalize with batch norm parameters of the regular network?

We added this experiment to the ablation study in Fig. 9 and found that this configuration does not work. It is very similar to the other, naive, configuration (Fig. 8) of using a target network + BatchNorm. We suspect that the same hypothesis could be used here, which is that the normalization parameters used from the live network do not take into account the on policy actions.

5. CrossQ Bounded Activations Ablation

Do you have an intuition for why CrossQ (black line) performs worse on HalfCheetah in Figure 7?

That is because the black-line curves in Figure 8 (previously Figure 7) are not CrossQ, they are only the SAC (tanh) configuration most resembling CrossQ. Figure 8 presents a controlled experiment using tanh activation functions everywhere, because tanh activations let us analyze the effect of the other components on SAC without the presence of target networks.

We hope this addresses the reviewers concerns. If the reviewer has no additional concerns, we would kindly ask them to increase the score.

4. Additional Environments, Baselines, and Ablations

I feel that the claim of “state-of-the-art” performance is a bit strong given that only 6 MuJoCo environments were tested, and CrossQ is a clear improvement only in 2 of them. The method has not been evaluated in other continuous-control problems nor against other potential baselines, so I would not necessarily agree that it is state-of-the-art. Nevertheless, the empirical results are still great.

We have toned down the claims within the paper by specifying more exactly that our claims are restricted to continuous-control tasks with fully observed state. In our original submission, we followed the prior work, REDQ and DroQ, and used the same benchmark suite, but we even extended the experiments to 6 environments rather than 4 as in the previous papers. In our updated submission, we included experiments on DeepMind Control Suite, which show the same trend in the improved sample efficiency provided by CrossQ. However, note that we had a limited time during the rebuttal to tune CrossQ on those environments, and better results may be achievable till the camera-ready submission deadline. We added:

1. evaluations on 6 additional Deep Mind Control environments,
2. a TD3 baseline (Fig. 5),
3. additional ablations (Fig. 9, 14, 15),
4. a version of CrossQ based on TD3 (Fig. 5).

We hope this demonstrates the more general applicability of our proposed method.

5. Why BatchNorm Works in CrossQ But Not in DroQ

At the end of the introduction, you say that your success with batch norm “contradicts” another paper [2] that did not find batch norm to work well. Why do you think you were able to achieve better results? Is it because you removed the target network, or is there another reason

Indeed, DroQ reports experiments with BatchNorm, but they find that it does not improve the performance. Unfortunately, the DroQ code release does not include BatchNorm, therefore, we cannot check the exact implementation. However, we hypothesize that the DroQ authors used BatchNorm in combination with target networks and did not account for the different batch statistics (Figure 4), essentially creating the setting which fails to train, as demonstrated in our ablations (Figure 9 (light blue)).

We were able to use BatchNorm successfully because we took care to use identical normalization moments for both (s, a) and (s’, a’) batches. Normalizing with identical moments is trivially done using a single combined-batch forward pass; this necessitates predicting both Q(s, a) and Q(s’, a’) through the same network, which by design cuts out the target network.

Reviewer 7EH3 suggested another way to normalize with identical moments, which does not entail target network removal: reusing the (s, a) BatchNorm moments from the “live” network within the target network. We added an experiment testing precisely this configuration in Fig 9, which performs very poorly.

We therefore attribute the superior performance of CrossQ to the synergistic combination of identical normalization statistics and target network removal.

6. Joint Forward Pass

Figure 2’s caption says, “this removes the computational need for two individual forward passes through the same network” in reference to batching the observations and next observations together. But this doesn’t actually reduce computational cost, does it? The same number of forward passes are being done either way.

We have rewritten this sentence in the paper to make it clearer. We intended it as a small side note and not as a claim that this is where CrossQ gets its efficiency from. It is true that the same number of operations would be performed in a bigger batch, though on a GPU, performing them in a single bigger batch may be slightly more efficient because this can be done within a single CUDA call instead of two consecutive ones. But it is by no means a crucial point, so we clarified it in the updated submission.

7. Q-Estimation Bias Evaluation Procedure

Could you explain the evaluation procedure in the Q-estimation bias experiment (Figure 6)?

We used the exact same procedure that was proposed in the REDQ paper. They define the Q-estimation bias as the averaged difference between the Q function estimate for the (s, a) pairs sampled at each timestep of an evaluation episode, and their corresponding true Monte Carlo return within that episode: $Q(s, a) - \sum_t \gamma^t r_t$. This Q-estimation bias is then additionally normalized by the magnitude of the Monte Carlo returns, to adjust for the effect of increasing returns over the course of training.

We thank the reviewer for a lot of thorough questions and suggestions for improvement. We incorporated all the feedback into the updated paper and provided our detailed answers below.

1. Generality of the Proposed Method

It would be exciting if this generalizes to other algorithms as well.

This is an interesting observation. We based CrossQ on SAC because the baselines REDQ and DroQ are also based on SAC. However, our concepts are directly applicable to other off-policy TD algorithms. For demonstration, we added a TD3 baseline, as well as a version of CrossQ based on TD3 to Figure 5. This combination shows the same effects in terms of increased sample efficiency, confirming that our proposed design choices regarding batch normalization and target networks can be applied more broadly and are not specific to SAC and its max-entropy RL setting.

2. How BatchNorm Was Used in DDPG

The paper claims it “show[s] the first successful application of BatchNorm in off-policy Deep RL.” However, it appears that DDPG [1] also used batch normalization 7 years ago in the exact same setting: off-policy deep RL for continuous control in MuJoCo. This means the titular contribution of the paper is not original.

It is true that the DDPG paper reported experiments with batch normalization. However, we would like to emphasize that ours is arguably the first “successful” application of BatchNorm in Deep RL. There are two crucial points:

1. In the DDPG paper, BatchNorm does not significantly improve the performance, and in the absence of the target networks even actively harms the performance (see Fig. 2 in https://arxiv.org/abs/1509.02971). Our paper is the first to use BatchNorm to effectively accelerate Deep RL training by an order of magnitude, similar to supervised learning.
2. The DDPG paper’s use of BN is limited: it relies on a unique architectural choice where actions are fused late into the critic, and applies BN only to early layers which appear before this fusion.

   “In the low-dimensional case, we [i.e., the DDPG authors] used batch normalization on the state input and [...] all layers of the Q network prior to the action input.” (Lillicrap et. al 2016).

This is a smart way to avoid dealing with the on- and off-policy action distribution mismatch in BN — one simply does not use BN with action-informed features — but is unfortunately only applicable to this specific late-fusion architecture, which is not commonly used.

Moreover, we found several online discussions that claim that BatchNorm actively harms training [1][2], and we could not find any open source implementation of DDPG using BatchNorm. Therefore, we argue that our CrossQ algorithm is effectively the first method that provides a working open source implementation of batch normalization in Deep RL that results in improved learning performance.

We added a clarification about the difference of our approach compared to DDPG in Sec. 3.2 “Design Choice 2: Using Batch Normalization”.

[1] https://github.com/keras-team/keras-io/issues/198

[2] https://www.alexirpan.com/2017/04/26/perils-batch-norm.html

3. Additional REDQ and DroQ Ablations

There is a lot of hyperparameter tuning that might be unfairly benefiting CrossQ. For instance, the network width for CrossQ is increased from 256 to 2048 units, an 8x increase. Adam’s momentum parameter was also reduced from 0.9 to 0.5. As far as I can tell, these hyperparameters were not tested for the baselines, so CrossQ has been over-optimized for MuJoCo. A fairer comparison would be testing the methods with the same network sizes and then selecting the best Adam hyperparameters for each of them.

We would like to note that CrossQ performs well even with a smaller critic network and Adam’s momentum 0.9, as shown in Fig. 9 by the “CrossQ (small)” line. Nevertheless, we added the requested ablations, running DroQ and REDQ with increased critic networks and smaller Adam’s momentum in Fig. 14 and 15. As can be seen, DroQ and REDQ perform the same or worse with these hyperparameters. Therefore, this ablation confirms that these parameters alone are not sufficient to improve upon the baselines, and the other design choices of CrossQ such as BatchNorm and removal of the target network are also important.

8. Policy Delay

Could you also explain what is meant by a policy delay of 3? How is this different from the UTD ratio?

First introduced in the TD3 paper [3], the policy delay prescribes the number of actor update steps that are being skipped for every critic update. With a policy delay of 3 in CrossQ, we still do one critic update per environment sample (i.e. UTD=1) and only update the policy once every third environment sample. The UTD ratio, as used in the REDQ and DroQ papers, prescribes the number of critic updates per environment sample.

[3] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 2018.

9. Minor Edits

We thank the reviewer for pointing these out and have corrected them in the updated version of the paper.

We hope this addresses the reviewers concerns. If the reviewer has no additional concerns, we would kindly ask them to increase the score.

Answer to enbH

We thank the reviewer for suggesting insightful additional experiments. We conducted the suggested studies and added the results in the updated paper. Below we discuss our observations. We hope our answer and the updated paper provide strong evidence in support of our original findings about the proposed CrossQ algorithm and increase the reviewer's score.

1. Papers Claims and Broader Analysis

The paper makes very broad claims in terms of proposing the new state of the art in both sample and compute efficiency - they might be justified, however only relatively few baselines (DroQ, REDQ, SAC) are used for comparison. For statements like this, a broader comparison might be required.

We have narrowed the claims in the paper by specifying more exactly that our results are restricted to continuous-control tasks with state observations. In our original submission, we followed the prior work, REDQ and DroQ, and used the same benchmark suite as them, but we even extended the experiments to 6 environments rather than 4 as in the previous papers.

As requested, we made the comparison broader in two ways:

• In our updated submission, we have now included 6 additional experiments on the DeepMind Control Suite, which show the same trend in the improved sample efficiency provided by CrossQ.
• Regarding the baselines, we further added:
  1. a TD3 baseline,
  2. additional ablations on the current baselines, and
  3. a version of CrossQ based on TD3.

We hope this demonstrates the more general applicability of our proposed method (see Figure 5).

2. Longer Training Runs

 if the superior efficiency were only attainable for suboptimal performances, that would be quite limiting and needs to be examined. Please clarify

We initially conducted all our experiments with a limit of 300k environment interactions, because the baseline REDQ and DroQ papers also use this limit. The focus is on sample efficiency on a limited budget of environment interactions, and the baselines are also computationally costly. However, in our updated submission, we do present longer runs of CrossQ in the Appendix in Fig. 13, to demonstrate that longer training runs allow for reaching higher final performance, e.g., on HalfCheetah, Humanoid or Walker-2D.

3. Action Distributions

visualize the distributions of sampled and policy actions

We have added the requested plot to the updated paper. Figure 4 now shows the difference in distributions between on-policy actions and those sampled from the replay buffer. In the Appendix, we also extended this visualization in time, showing how both distributions evolve — and remain different! — over the course of a training run.

We hope this addresses the reviewers concerns. If the reviewer has no additional concerns, we would kindly ask them to increase the score.

4. CrossQ on Vision-based RL Tasks

Also, this paper does not include image-based settings. There are tons of efficient RL works recently on image-based settings and some of the techniques (state augmentation [1, 2] and auxiliary losses [3, 4]) can be also applied to state-based settings. I think it is worth discussing the recent progress of efficient RL in the related work session or adding baselines for comparison.

We agree that a thorough evaluation of CrossQ on vision-based tasks would be very interesting; however, we consider this extension as being beyond the scope of the present paper. Efficient image-based RL requires multiple ingredients, as noted by the reviewer, and a careful study of CrossQ in tandem with these would constitute a separate publication.

In the current paper, we focus on establishing CrossQ as a state-of-the-art algorithm for continuous control from state observations, closely following the setting of REDQ and DroQ. We do agree that much like our use of BatchNorm, the techniques mentioned by the reviewer are also strong tools for boosting RL efficiency; we have added the suggested references and highlighted the extension of CrossQ to vision-based tasks as the next step for future work.

5. Are Wider Critic Networks Crucial for CrossQ Performance?

To me, I think using wider critic networks played a pivot role in the performance boost, especially in humanoid-based environments. Can you draw SAC+wider networks to Figure 7?

We note that even without the wider critic network, CrossQ delivers a strong performance, as shown in Figure 9 in the updated paper by the magenta curve corresponding to “CrossQ (small)”. We have added the requested SAC+wider network “SAC wide Net” ablation study to Figure 9 (yellow curve). There, we observe that SAC does not get a significant improvement simply from adding wider networks, indicating that it is the combination of CrossQ’s design choices that is important, and not only the wider critic networks alone.

We hope this addresses the reviewers concerns. If the reviewer has no additional concerns, we would kindly ask them to increase the score.

We wish to thank the reviewer for their comments and positive feedback. We believe we have been able to address all their points in the following paragraphs.

1. Insights and Ablations

While the empirical results are compelling, the paper lacks theoretical analysis or justification for the success of CrossQ. Providing insights into why the proposed modifications lead to improved performance would enhance the paper's depth and contribute to a more comprehensive understanding of the algorithm.

We added further insights into why the proposed modifications lead to improved performance in several places in the paper:

1. we added Fig. 4 that show the difference in the action distributions from the replay buffer and from the policy, highlighting the importance of using batch normalization in the proposed manner;
2. in Fig. 9, we added an ablation experiment running SAC with both BatchNorm and Target Networks (as suggested by reviewer 7EH3), showing that our design choice of removing the target network is important for the superior performance of the proposed CrossQ algorithm;
3. we added Fig. 14 and 15, providing ablations on DroQ and REDQ with elements of CrossQ such as wider critic networks and $\beta\_1 = 0.5$, showing that these changes alone are insufficient to boost their performance, and it is really the combination of the components in CrossQ that is responsible for the strong learning performance.

Thus, we tried our best to provide intuitions and establish general facts about the proposed design choices. Nevertheless, we admit that a deeper theoretical investigation would be very illuminating, despite being a difficult task. We hope that our paper may stimulate interest in studying the effects of different design choices in Deep RL algorithms.

2. Strengths and Weaknesses of CrossQ

Can you elaborate on the specific scenarios or domains where CrossQ might outperform other algorithms significantly? Are there any limitations or challenges in which CrossQ may not be the most suitable choice?

This is a very good question!

In general, we believe CrossQ should outperform algorithms which use low UTD ratios and use no special techniques to accelerate convergence. E.g. CrossQ outperforms SAC and TD3 significantly, but is comparable in efficiency to REDQ which emphasizes making the most out of the available data.

In addition to the potential limitations we’ve already noted in the conclusion section, we believe CrossQ may inherit any challenges that BatchNorm introduces. For example, models with BatchNorm are known [1] to struggle in training with very small minibatch sizes. BatchNorm uses minibatch statistics to normalize data, and while minibatch noise is regarded as a regularizer, it has also been implicated in sometimes harming optimization stability [2]. BatchNorm-equipped training is also sensitive to non-i.i.d batches, and could possibly struggle with non-stationary data sampling schemes such as Prioritized Experience Replay [2]. Such limitations in BatchNorm have been addressed with very simple fixes like Batch Renormalization [4], which might also benefit CrossQ in the future.

[1] https://arxiv.org/abs/1708.02002

[2] https://proceedings.mlr.press/v89/lian19a.html

[3] https://github.com/yhenon/pytorch-retinanet/issues/24

[4] https://arxiv.org/abs/1702.03275

3. Additional Environments and Evaluations

The experiments are only conducted in one continuous control domain (MuJoCo). It is dangerous to draw conclusion of sample efficiency only from one experimental domain.

We followed the prior work, REDQ and DroQ, and used the same benchmark suite. In fact, even in our original submission we extended the experiments from 4 to 6 environments, adding the hard HumanoidStandup task. Following the reviewer's suggestions, in our updated submission we have doubled the number of environments by adding 6 DeepMind Control Suite tasks. We observe the same trend of improved sample efficiency provided by CrossQ (see Fig. 10). We hope that this addresses the reviewer's concern about the applicability of CrossQ to other environments.

That should indeed say next_obs. Thanks for pointing this out! We have fixed it in the newest update.