The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning

We would like to thank the reviewer for their obvious extreme care in providing us with feedback. We are encouraged by the reviewer’s comments that the paper “should be accepted,” "will be an important contribution to the community,” and are grateful for their detailed comments which will help us improve our paper.

---

Weaknesses - Formatting, punctuation, use of subsections, etc.

We thank the reviewer for their detailed feedback on the organization and formatting of the paper. Having revisited the paper, we completely agree with the reviewer’s comments. We will use the extra page allowance to make the paper feel less crowded, and will make all of the suggested changes (use of italics, commas and colons, subsections, etc.) ahead of the camera-ready submission. We are grateful for the care taken in providing a detailed list of suggested edits, and would like to thank the reviewer again for their time in helping us improve the paper.

Q1 - Implementation details of Q-function initialization

We use the default initialization scheme (Kaiming uniform). We thank the reviewer for asking this and will add this implementation detail to the paper.

Q2 - Effect of initializing the Q-function at zero

[Please see the Rebuttal PDF.]

The reviewer asked whether the edge-of-reach problem could be solved by simply initializing the Q-function to be zero for all inputs. This is a good question and gets to the heart of the pathological issues involved in the edge-of-reach problem. Initializing a network so that its output is zero given any input is non-trivial. (The obvious way to do so would be to initialize all weights to zero, but the network would then not train.) Thus, we instead we consider what would happen if we initialize so that all Q-value outputs are very close to zero.

As the reviewer commented, in this case we might expect the edge-of-reach problem to be avoided since edge-of-reach states cannot then be a source of overestimation if their Q-values are significantly less than the optimal Q-values. However, as shown in the histograms in the rebuttal PDF, the standard NN initialization that we use already in all our experiments already results in all Q-values being very close to zero at initialization ($<1e-2$). Moreover, all these values are much lower than the optimal Q-values ($\sim 5e+3$) and much lower that the Q-values reached over training ($>1e+10$) (see Figure 3). Yet we still observe the edge-of-reach problem. This means the above line of reasoning does not hold.

Instead what happens is that, when values at within-reach states are increased, the use of NNs as function approximators means that the Q-values at nearby edge-of-reach states may also increase. Q-values at edge-of-reach states are therefore not guaranteed to remain small over training, and hence can become a source of overestimation. Moreover, overestimated Q-values lead to other nearby Q-values also being raised, which can then lead to the original Q-values being raised further, thus setting up a pathological positive feedback cycle whereby Q-values increase unboundedly over training (as seen in Figure 3). This is investigated and explained more thoroughly in [1] and [2] in the context of the analogous model-free offline problem (where function approximation means out-of-sample state-actions become a source of pathological overestimation).

We thank the reviewer for this important question. We will use the camera-ready extra page allowance to include some more of the detail from [1] on the role function approximation and this pathological positive feedback loop.

[1] - Aviral Kumar, et al. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, 2019.

[2] - Sergey Levine, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.

Q3 - Clarification on the $x$-axis of Figure 5

The $x$-axis of Figure 5 shows variance over RAVL’s Q-ensemble. Since RAVL takes the minimum over this Q-ensemble (see Equation 2), the penalty being added is effectively proportional to the Q-ensemble variance, hence why we labeled the $x$-axis as “RAVL’s effective penalty.” We described this briefly in the caption, but we will make this clearer in the camera-ready version.

---

We thank the reviewer again for their careful review of our submission. The question about initializing the Q-function at zero was particularly insightful. We are also extremely grateful for the feedback about formatting and the detailed list of suggested edits. We agree with these and will make the suggested changes. Thank you for your help in improving our paper. Please let us know if you have any more questions or comments as we would be more than happy to discuss further.

We would like to thank the reviewer for their careful feedback on our submission.

We are grateful that they appreciated our “thorough investigation, theoretical analysis, and empirical evidence to support [our] claims” and “robust performance across benchmarks.”

We found their questions particularly insightful (see below).

---

W1.1 - Performance on D4RL

The reviewer highlights that, on the D4RL benchmark, we only match SOTA rather than surpassing it. Below we discuss why this is expected and why we believe this does not affect the key contribution of our paper.

In Section 6.5 we describe why existing methods can sometimes be made to work despite misunderstanding the underlying issues in offline RL. (tl;dr: If there is a lucky correlation between dynamics uncertainty and EOR states, then the dynamics penalties of existing methods can indirectly address the true EOR problem.) For the common D4RL setup (on which many existing methods have been developed) this lucky correlation holds (see Figure 6), meaning existing methods can be made to work well, and we would not necessarily expect RAVL to beat them. (Further, the majority of SOTA scores are close to, or already exceed, expert level $>100$.)

In general, however, this lucky correlation is not guaranteed, and the indirect approach of existing methods means we may expect them to be very brittle to dynamics model changes. Indeed, in Figure 1 of the rebuttal PDF, we show that existing methods completely break down if the dynamics model is changed slightly. RAVL, by contrast, is much more robust due to directly addressing the true underlying EOR problem (see Figure 1 in the rebuttal PDF).

We believe the correction of a significant misunderstanding, and RAVL’s consequent much-improved robustness are highly valuable contributions to the community.

W1.2 - Performance on more complex and challenging benchmarks

We thank the reviewer for asking about more complex and challenging benchmarks. We would like to highlight our additional results on the pixel-based V-D4RL benchmark in Appendix F (moved to the Appendix due to space limitations).

Here RAVL represents a new SOTA, giving a performance boost of more than 20% for some environments. These results are particularly notable as the pixel-based setting means the base algorithm (DreamerV2) uses model rollouts in an imagined latent space (rather than the original state-action space as in MBPO). The results therefore give promising evidence that RAVL is able to generalize well to different representation spaces.

We will use the camera-ready extra page to feature these results in the main paper. We hope this answers the reviewer’s concern with a much higher-dimensional and more challenging benchmark.

Q1 - What happens if we increase the rollout length as dynamics accuracy improves?

We thank the reviewer for this interesting and insightful question. Tuning the rollout length as suggested is indeed what we tried first.

In the limit of perfect dynamics accuracy this of course works, since rollout length can be increased to full horizon and the procedure becomes online RL. For intermediate dynamics model accuracies, however, we experimented with this extensively but found we were unable to get existing methods to work.

Our intuition for why this is is as follows:

Consider a 2D environment like in Section 4, and imagine we have a dynamics model for which error and uncertainty is significant to the left, and negligible to the right. The model being imperfect means rollouts will need to be truncated. (All existing methods heavily truncate rollouts e.g. $k=1,5$ to avoid compounding errors, even with dynamics uncertainty penalties). However, rollout truncation means we now have EOR states.

The question is: Is it possible to tune dynamics uncertainty-based methods to work in this setting?

Uncertainty penalties can successfully penalize the EOR states to the left. However, it is impossible to get uncertainty penalties to target the EOR states to the right (since model uncertainty to the right is negligible).

This illustrates that we can have dynamics models for which it is theoretically impossible to get uncertainty-based methods to work, even allowing for tuning the rollout length. (RAVL, by contrast, would be expected to work in this setting since it directly ensures all EOR are targeted.)

The much stronger reliance on the dynamics model for uncertainty-based methods compared to RAVL (as highlighted by this setting) fits with our empirical observations of existing methods being much more brittle to changes in the dynamics model. See for example our new results (Figure 1 of the rebuttal PDF), in which we find that RAVL is significantly more robust, both when model accuracy is increased and decreased.

We thank the reviewer for asking this. We found it to be a very interesting and insightful question and would be more than happy to discuss further.

Q2 - Does the "edge-of-reach" hypothesis exist in many online RL scenarios?

Thank you for another insightful question!

The trajectory cutoff, for example $H=1000$ for MujoCo, can indeed be viewed as a truncated horizon. This means the EOR problem could in theory be present in online RL. We anticipate, however, that $H=1000$ is sufficient for the agent to be able to reach all possible states within this horizon, meaning that the set of EOR states will be empty and the EOR problem will not be present.

As briefly mentioned in the limitations section, however, we are interested in investigating whether truncated horizons could be used in some way in online RL to harness the EOR effect and induce an implicit exploration bias.

---

We thank the reviewer again for their valuable feedback and insightful questions. We hope we have addressed all of your concerns. Please let us know if you have any remaining questions. If we have been able to address your concerns we humbly ask if you would recommend acceptance.

We thank the reviewer for their careful review of our submission.

We are encouraged that the reviewer found the problem to be “interesting,” and the experiments “comprehensive, well-designed”, and are grateful they appreciated that the work is “the first to formally investigate the problem”. We have addressed their questions and comments below.

---

W1 - Combining RAVL with dynamics uncertainty penalization

The reviewer asked for support of the claim that "RAVL can be easily combined with appropriate dynamics uncertainty penalization.”

In terms of implementation: this is extremely simple since dynamics penalization is typically a change to the dynamics model’s reward, while RAVL’s value pessimism is a change to the agent’s value function update.

In terms of empirical evidence: we refer the reviewer to the V-D4RL experiments. As stated in Appendix C.3, for the RAVL results we add RAVL’s value pessimism in addition to the dynamics pessimism already used in the base procedure. As such, the V-D4RL results are an example of combining RAVL’s value pessimism with dynamics uncertainty penalization, exactly as requested. This provides evidence that RAVL’s value pessimism is effective even when the model is less accurate and dynamics uncertainty is necessary. We will make the details in Appendix C.3 more prominent in the camera-ready submission.

W2 - Formatting e.g. italics affecting readability

Having looked over the paper again we completely agree with this. We will make sure to refine our use of italics and bold text etc. ahead of the camera-ready submission. Thank you for bringing this to our attention.

Q1 - Why does RAVL not benefit from reduced dynamics errors according to Table 1?

Thank you for this question.

Section 3.4 claims and gives evidence that dynamics model errors are already negligible for the main D4RL benchmark (and that issues are instead caused by the edge-of-reach problem).

The trend referred to (namely that reducing dynamics errors does not affect performance) is therefore exactly what we would expect, and the results in Table 1 can hence be seen as further evidence that (contrary to common understanding) dynamics errors contribute negligibly to the issues seen in offline model-based RL.

Q2 - Results of RAVL's performance with different dynamics model accuracies

[Please see the rebuttal PDF.]

We have run additional experiments (4 seeds), and have added results for RAVL to the original Figure 1 as requested (please see the new Figure 1 in the rebuttal PDF). As expected, RAVL’s performance remains strong as dynamics model accuracy increases. Furthermore, RAVL appears to be significantly more robust than dynamics uncertainty-based methods as dynamics model accuracy is reduced.

We sincerely thank you for asking for this. These results significantly strengthen our paper and we are excited to add them to the camera-ready submission.

---

We thank the reviewer again for their invaluable feedback. We hope we have addressed all of your questions and concerns. Please let us know if there is anything remaining as we would be more than happy to discuss further. If we have been able to address your concerns, we humbly ask if you would consider raising your score.

We would like to thank the reviewer for their valuable feedback on our submission.

We are particularly encouraged by their appreciation of the care we took with the empirical investigations, in particular with the comments that we provide “a very convincing set of evidences for the paper's claim,” and “set a good example for this kind of empirical research”.

---

W1/Q1 - “necessity of the detection of an edge-of-reach state”

The suggestion of removing the explicit “detection” step is in fact exactly how RAVL is implemented (see Equation 2). We just chose to describe it as (A) detecting and (B) penalizing edge-of-reach states to provide intuition of what RAVL is effectively doing. Thank you for this question. We will make it clearer in the paper that this two-phase description is just for intuition purposes.

W2 - Position in literature and additional references [1] and [2]

Thank you for highlighting this. We have referenced [1] briefly in the related work section already, but we will use the camera-ready extra page allowance to more thoroughly discuss value pessimism in online RL, including the references [1] and [2] you suggested.

W3 - Effect of edge-of-reach states in online MBRL

Thank you for this question. Yes, we are also interested in extending this work to online MBRL. One key distinction here is the pool of initial rollout states. In offline MBRL this is fixed over the whole of training ($s_0 \sim \mathcal{D}_\text{offline}$), while in online MBRL this changes as the agent collects additional trajectories online. The fixed initial state distribution is an important aspect of edge-of-reach states (see Definition 1). There may, however, be a somewhat analogous notion of edge-of-reach states in online MBRL. As briefly mentioned in the limitations section, we would be interested to investigate whether this could be used (or perhaps already plays a role) as an implicit exploration bias in online MBRL, and believe this could be an exciting direction for future work.

W4 - Future referencing

Thank you for bringing this to our attention. We agree that the current future referencing may be confusing. Your pointer of instead notifying the reader that “more detail will appear in the future” rather than asking them to jump, is extremely helpful and we will definitely make this change. We plan to also use the camera-ready extra page allowance to remove future referencing where possible.

---

We thank the reviewer again for their valuble feedback. The pointers (in particular about style of future referencing) are really helpful. We hope we have addressed all of your questions. Please let us know if any further discussion would be useful as we would be more than happy to answer additional questions.