Thank you for your review.

1. Better present the theoretical result

We have now phrased the result as a proposition in the revised paper. Thank you for your feedback.

2. Hyper-parameter table

We have put a hyper-parameter table in the appendix. Thank you for your feedback.

3. Importance and best practices for hyper-parameter $\lambda$

We have added a sensitivity analysis on $\lambda$ for the SAC + Mujoco experiment and we describe how we ended up choosing $\lambda = 0.1$ for that particular experiment which hopefully provides some helpful insight. Because FQE diverges relatively quickly in that setting with high update rates of the target, it ended up relatively easy to set a sensible parameter for that experiment. We discuss this in more detail in this new section of the appendix.

4. Other datasets than Mujoco

We have launched some additional experiments on LunardLander-v2 and MinAtar tasks on DQN. Please refer to the new appendix for more details. These preliminary results indicate that the BE term can also be efficient in stabilizing DQN in high target network update regimes.

5. Integration with other techniques such as ensembling, higher update-to-data ratio, double actors, etc.

There has been a large amount of prior work aimed at improving the sample efficiency of deep RL algorithms. While it is difficult to make definite statements about how this particular auxiliary loss would combine with existing ones, we would like to point out that the mechanisms by which sample efficiency in our case is improved is very unique. Indeed, to the best of our knowledge this regime of high update rate of the target network remains rarely studied as a way of speeding up deep RL. We have shown that it can be impactful on Mujoco tasks and with the new baseline added in the revision, that other modeling-based auxiliary losses are not as effective as the proposed BE loss in preventing divergence of FQE in this regime. Because the properties of our approach are rather unique, we believe there is a good chance that it is complementary with existing methods. Of course successfully integrating these different methods is not trivial but can be an interesting future direction.

Thank you for your review.

1. $L(\phi)$ follows prior work

There seems to be some confusion on what $L(\phi)$ is and what part of our work is novel. To the best of our knowledge, no other prior work proposes to minimize $L$ for non-linear policy evaluation. If the Reviewer has references showing otherwise, we would be glad to cite them.

We would also like to clarify that $L$ is a Bellman error, where the linear part is the minimizer of the projected Bellman error. $L$ itself is not a model learning loss but can be shown to be bounded by one. To the best of our knowledge this insight is also novel and is the main motivation for our research question: if the usual model learning auxiliary losses are upper bounds to a Bellman error, why not minimize directly the Bellman error as an auxiliary loss?

2. Section 3 unrelated to the rest of the paper

The main point of Section 3 is to show the relationship between existing work on regularising policy evaluation algorithms with auxiliary modeling losses and the proposed Bellman error $L$. The conclusion of Section 3 is not “let us do something else” but instead, why not minimize the Bellman error $L$ directly instead of the upper bounding model learning losses of prior work.

Optimizing $L$ directly is challenging due to the numerical instability of computing the LSTD solution [1]. The practical algorithm is to use instead the current $w$ from an FQE procedure running concurrently as an approximation to the LSTD solution and use this $w$ for computing $L$. While different from computing $L$ exactly, we believe this still lets us investigate the research question raised by Section 3 which is about the merits of using a Bellman error instead of the environment modeling error.

We hope this clarifies better the relation between Section 3 and the remainder of the paper. Otherwise we would really appreciate some additional feedback to improve the presentation of the paper.

Now because $L$ combines both a Bellman error and the minimizer of the projected Bellman error, we believe the refresher in Section 2 is still useful to fix the terminology.

[1] Scherrer; Should one compute the temporal difference fix point or minimize the bellman residual? the unified oblique projection view. ICML10

3. Is $10^6$ time-steps not the norm for MuJoCo?

Prior work already considered shorter total time-steps. E.g. REDQ [CITATION] has Mujoco experiments with 120 to 300K time-steps. In our setting the improvement in sample efficiency is most apparent around the 250K time-step mark and our plots are meant to focus on this region.

4. Lack BC baseline for the SAC + MuJoCo experiment

Thank you for your suggestion. We have now added this baseline.

5. How is $\lambda$ defined

Lambda is a hyper-parameter. We have now included additional plots in the appendix to show the impact of this hyper-parameter.

Thank you for your review

1. Sensitivity to hyperparameter $\lambda$

Thank you for your feedback. We have run additional experiments on the SAC + Mujoco setting with 4 additional values of $\lambda$ on the previous 4 Mujoco tasks and averaged over 20 seeds. Results and discussion can be found in the appendix. Please let us know if this addresses your concerns.

2. Will the method benefit from more gradient updates

The way this paper aims at improving sample efficiency is by recomputing the target more often. This leads to more frequent applications of the Bellman operator and lets the rewards propagate faster through the chain of state-action pairs. This shows well in the policy evaluation experiment where we can compute the distance to the true Q function and has tangible benefit when solving RL tasks as shown in the SAC + Mujoco experiments. To the best of our knowledge, no other work has investigated improving sample efficiency by making FQE more robust to faster target value recomputations. We think because its properties are unique, it is more likely that our work can integrate well with prior improvements of SAC/deep RL such as increasing the Q-function ensemble size or performing more gradient steps.

3. Computation overhead

In the SAC + Mujoco experiment, the execution time of DouBel was about ~1.2 times the execution time of vanilla SAC, while DouBel with bias correction was ~1.5 times the execution time of vanilla SAC, because of the extra V-value network. We have added this to the appendix, thank you for pointing it out.

4. Will the proposed method lead to better long-term performance

We have not observed better long-term performance in the Mujoco task. We think the main benefits of our methods are in the initial speed-up provided by a more frequent application of the Bellman operator. That is why our results focus on the 0-500K range for the SAC + Mujoco task.

5. Efficiency compared to other approaches and possible combinations

Methods such as AQE can be about 200 times slower than SAC, because it has a 10 times larger ensemble size and uses 20 times more gradient steps per sample. Under a fixed computational budget this might not even be more efficient than vanilla SAC. Of course this all depends on the environment and if the cost of interaction is very high (e.g. robotics) it might still be beneficial to focus on sample efficiency first and foremost. As we have shown in the new baseline requested by Reviewer JxN6, auxiliary losses such as those predicting next state features do not stabilize FQE in the high update rate of the target regime, and hence the advantages of our methods are unique. Because of that we believe it can complement quite well prior work towards the goal of developing increasingly more sample efficient deep RL algorithms. Of course merging all these approaches is not necessarily trivial but can be an interesting future direction.

Thank you for your review.

1. What are $m_r$ and $M_\Phi$

$m_r$ and $M_\Phi$ define linear models in $Phi$ that minimize the squared error to the reward $\cal R$ and the next state features $\Phi’$. We have expanded the explanation in page 5. We hope it is clearer now.

2. Clarifying what are (aggressive) target network update rates

We have updated Sec. 2.2 to clarify what target update rates mean. Please see the bolded text.

As for what aggressive update rates are, it depends greatly on the task. For example, tuned hyper-parameters in stable-baseline 3 suggest an update rate of 10 for CartPole but 1000 for Atari.

In our SAC + Mujoco experiment, we consider the update rate of 10 as aggressive since vanilla SAC, without the proposed regularizer, ‘diverges’ in many cases.

For the policy evaluation task, at the fastest update rate of 1, FQE(1) does not diverge but adding our regularizer has a big impact on the distance to the true Q function, whereas for an update rate of 20, DouBel(20) and FQE(20) are about the same in performance and convergence profile.

3. Adding a simple experiment

We experimented with small scale tabular environments with rich observations (i.e. the MDP is finite and small but the learner observes high dimensional vector valued states). However, in these simple MDPs, Neural FQE was converging very fast to the true Q function even with a target update rate of 1. These simple MDPs were thus not able to illustrate the challenges of policy evaluation with neural function approximators that one can experience on more complex MDPs.

4. DouBel(20) and FQE(20) have much lower loss than DouBel(1) and FQE(1)

Yes, experiments in 6.1 show that if the computational budget allows it, the slower update rate of 20 is better. However, the learning curve illustrates well how slow policy iteration can be in this regime. Because the targets are updated less frequently, the Bellman operator is applied less often and it takes more time to reach a fixed point. To give an idea of the scale of the X axis for the policy evaluation experiments: we perform up to 10K gradient steps, but an algorithm such as PPO on Mujoco performs typically only 320 gradient steps per iteration for its policy evaluation! Even after 10K gradient steps, there are still four Mujoco datasets where DouBel(1) outperforms DouBel(20). Given how computationally expensive deep RL can be, we believe it is still important to consider improving neural FQE through this faster update of the targets. One can always revert to slower update rates once the error reduces enough. The claim of the paper still holds that these faster update rate regimes do not seem helpful with vanilla FQE but become interesting when equipped with our proposed regularizer.

5. DouBel always lower than FQE

In the policy evaluation experiment, both algorithms performed about the same for the update rate of 20. For the SAC experiments, we have included a new parameter sensitivity analysis and it shows that a too high $\lambda$ can be detrimental to performance. Some care should definitely be taken in tuning it, although the value of $\lambda=0.1$ seems to work well for Mujoco tasks and also the new DQN experiments we added to the paper.

6. Relation between theoretical results and more aggressive update rates

The theoretical results relate the usual auxiliary losses used in prior work based on modeling the reward and next state features with the proposed loss $L(\Phi)$. It is not about the learning dynamics of neural FQE. A hand-wavy argument could be that Bellman error minimization is stable because it is a true gradient descent method (unlike FQE), and adding an explicit BE term seems to be effective at avoiding gradient updates of the feature function that would blow-up the Bellman error. However, the only validation of this property in the paper is empirical.

7. Difference between Figures 6, 7, and 8

As can be seen in Table 1, the policy evaluation experiments are done on three datasets for each of the four Mujoco tasks corresponding to different performance levels of the data generating policy. Figures 6, 7, 8 show the learning curves on four Mujoco tasks for each of the three performance levels.

Before answering individually to each reviewer, we would like to recap the changes of the revised submission following reviewers’ feedback. 1- A new sensitivity analysis experiment on the SAC + Mujoco setting has been added to analyze the effect of $\lambda$ on DouBel, as requested by reviewers BJvN and X6y6. 2- A new baseline SAC + BC has been added as requested by Reviewer JxN6 3- Experiments on different environments as requested by Reviewer X6y6 using this time DQN as a base algorithm.

We have also made several adjustments to the presentation following the feedback of the reviewers. We believe that the incorporation of the reviewers’ feedback has significantly improved the paper and we thank them again for their careful consideration.