RAVL: Reach-Aware Value Learning for the Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning

We would like to thank the reviewer for their time. We are pleased that you found our experiments “interesting and insightful” and that the paper was “well-written and easy to follow”.

(Weakness 1) Does RAVL actually target edge-of-reach states?

This is a good question and gets to the crux of our method. Please see part A of our general response where we have included a detailed step-by-step walk-through of the intuition of why we can expect our RAVL approach to detect and penalize edge-of-reach states. We would be more than happy to provide further explanation, so please let us know if it is still unclear.

(Weakness 2, part 1) What starting states are being sampled in Definition 1?

Thank you for raising this. In hindsight, the notation in Definition 1 is slightly confusing. In Definition 1 we used $\mu_0$ to denote the starting state distribution of the rollouts (i.e. the distribution of $\mathcal{D}_{\text{offline}}$) as opposed to the initial state distribution of the environment. Thus, the edge-of-reach definition here is the correct one for offline model-based RL, and matches the one you describe. We apologize for causing confusion here and have adjusted the notation in Definition 1 to make this clearer.

(Weakness 2, part 2) Analysis of the Q-function update

Thank you for raising this. We provide this analysis as an illustrative example of how errors can be propagated from edge-of-reach states. We agree with the reviewer that in practice, model rollouts are appended to a buffer and training is performed on mini-batches of this data. However, as demonstrated empirically, the same optimization dynamics are likely to occur over the course of training, with edge-of-reach states not receiving any updates and errors from them propagating to all states. We have further clarified this in the paper after Propositon 1.

(Weakness 3) Empirical performance of RAVL

Thank you for raising this comment, however, we respectfully disagree with the assessment that the performance of RAVL is not strong, as it is state-of-the-art in 5/9 of the environments we evaluated on. In addition, we are pleased to be able to report that RAVL gives significant performance boosts on the more complex V-D4RL benchmark (please see part B of our general response). We also note that SOTA scores on the D4RL benchmark are likely saturated, already being significantly over 100 (normalized expert level) on many of the environments. As such, future work is unlikely to be able to demonstrate substantial improvements on this benchmark.

More importantly, however, we have demonstrated that MOBILE (the only algorithm that beats RAVL), along with many other prominent offline model-based algorithms, is derived based on significantly incomplete theoretical foundations (and we explain why they work despite this). It is the new insight, and the resulting much more complete understanding of offline RL, that we view as the main exciting new contribution of our work. We anticipate this should be hugely valuable in the future development of robust, strong-performing offline RL methods, potentially also resulting in new improved ways of tackling the critical, previously overlooked edge-of-reach problem.

(Question 1) Typo in Equation 3

Thank you for spotting the typo in Equation 3. We have now fixed this in the updated paper.

Thank you again for your insightful comments and questions. Spotting the typo and your feedback on the notation in definition 1 were especially helpful in improving the paper. We hope our responses were able to address any concerns and questions you had and convince you of the contribution of our work. Please let us know if you have any remaining concerns or if anything is still unclear - we would be more than happy to discuss further. If we have been able to address your concerns, we humbly ask if you would consider raising your score.

Dear Reviewer 34p6,

Thank you again for taking the time to review our paper. As author-reviewer discussion period is about to end (22nd Nov AOE, approximately two days), we were wondering whether our clarifications were able to address your questions and clear up any misunderstandings about our algorithm and the formalization in the paper.

Please let us know if you have any remaining questions or concerns. We hope that we have addressed your queries, and if so, would appreciate it if you would consider revising your score.

Best wishes,

The authors.

Dear Reviewer 34p6,

The OOD (high uncertainty) states that RAVL detects are with respect to $D_{\text{rollouts}}$ and not $D_{\text{offline}}$. Therefore, the scenario the reviewer describes could not happen and RAVL specifically targets edge of reach states which are by definition OOD with respect to $s\in D_{\text{rollouts}}$. This allows us to use existing Q-uncertainty methods but with respect to the model rollout distribution and not the original offline data distribution, and is the key novelty of our method.

Please let us know if we can help clarify further.

Best, The authors.

We would like to thank the reviewer for their time in reviewing our submission. We are pleased that you found that our paper presented an “intriguing finding” and are encouraged by your comment of it being “meticulously written and excellently structured.”

(Weakness 1) Influence of the paper

Thank you for this question. We strongly believe (like reviewer y8bf) that the analysis in our paper has the potential to have a substantial impact on future work in this area.

The main contribution of our paper is to expose and explain a significant gap in the current understanding of the issues underlying offline model-based RL. This current incomplete “dynamics model error”-based view forms the basis for many current offline model-based algorithms, such as MOPO [1], MOReL [2], and MOBILE [3]. Combined, these works alone have a total of over 1000 citations and have influenced a whole sub-field of offline model-based RL. Our paper demonstrates a significant misunderstanding in these works, provides and verifies an explanation (the edge-of-reach problem), as well as providing a solution (RAVL). We believe the resulting much more complete picture of offline model-based RL should be hugely valuable in the development of more accurately motivated and stronger-performing offline algorithms.

Furthermore, our detailed experiments show the edge-of-reach problem is not just an issue in theory, but that it is actually very prevalent in practice (on the simple environment, on the standard offline D4RL benchmark (see Table 1 and Figure 5), and even in latent space models (see our new results in part B of our general response).

Finally, as discussed in Section 6 and Appendix A, our work provides a unified view of model-based and model-free approaches, which we hope will enable more ideas from model-free literature to be directly applied to model-based algorithms (as we have done with RAVL), and we anticipate that our work may inspire further future connections to be drawn between the two subfields of offline RL.

(Weakness 2) Evaluation on harder environments

Thank you for raising this concern. We are pleased to present additional state-of-the-art results on the more challenging pixel-based V-D4RL benchmark (please see part B of our general response).

(Weakness 3) Simplicity of our algorithm

Thank you for noting that the approach in RAVL is quite simple. We agree, and we see this as a strength. Our aim was to make as few modifications to the base algorithm as possible, and our unified view of model-based and model-free approaches (introduced in Section 6 and Appendix A) enables us to directly take a robust out-of-the-box solution from model-free literature and apply it to solve the edge-of-reach problem. We found this worked extremely well and found no need to add anything to complicate it.

(Question 1) How was Figure 3 generated?

Thank you for raising this question. It has brought to our attention that the caption and labels in Figure 3 were unnecessarily unclear and we have edited them in light of your comment. In short: our construction of the simple environment allows us to exactly define whether a state is edge-of-reach or not (checking whether it is within fixed bounds). We are therefore able to compare the Q-ensemble variance for edge-of-reach vs. not edge-of-reach states. Figure 3 shows a histogram of the Q-ensemble variance (RAVL’s effective penalty) for all the state-action pairs evaluated over training, separated into whether the state is edge-of-reach (orange) or not edge-of-reach (ie. within-reach, blue). It shows that, as we hoped, the Q-ensemble variance (RAVL’s effective penalty) is significantly higher on states which are edge-of-reach, thus verifying that our RAVL method is effective at detecting and penalizing edge-of-reach states. (In part A of our general response we reiterate the intuition on how RAVL is able to do this.)

We would once again like to thank you for providing insightful feedback on our submission that helped us improve our paper. We hope our responses have clarified any questions you may have had. Please let us know if you have any remaining concerns or queries as we would be more than happy to discuss further. If we have been able to address your concerns, we humbly ask if you would consider raising your score.

[1] MOPO: Model-based Offline Policy Optimization Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, Tengyu Ma. NeurIPS, 2020.

[2] MOReL: Model-Based Offline Reinforcement Learning Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims. NeurIPS, 2020.

[3] Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, Yang Yu. ICML, 2023.

Thank you for the author's response, which basically solves my questions. Therefore, I tend to recommend this paper to the ICLR community. However, the "highlight" may be a bit too strong for this paper, so I prefer to keep the original rating (6).

We would like to thank the reviewer for their time in reviewing our submission. We really appreciate that you have taken the time to understand what we are trying to show, and are encouraged that, like us, you believe this work “is likely to have a substantial impact on future algorithmic work in this area”.

(Weakness 1) Evaluation on harder environments

Thank you for raising this concern. We have run RAVL on the more challenging pixel-based V-D4RL benchmark and are pleased to report new state-of-the art results (please see Part B of our general response).

(Question 1, part 1) Can the number of critics be reduced?

In our original submission, we stated that RAVL required N=50 critics to achieve sufficient edge-of-reach awareness on the hopper environments (while N=10 was enough for all the other environments). We are pleased to report that we have run additional experiments and have found that N=10 is mostly sufficient for hopper as well, with only hopper-medexp needing N=30, meaning that all but one environment uses N=10.

(Question 1, part 2) Did you try adding an explicit penalty to states at the end of rollouts as an alternative approach to the edge-of-reach problem?

Yes, this was an idea we had initially as well! We considered this for quite some time, but from our experiments it did not seem to work well. Our intuition on why this may not work is: Final step rollout states may not actually be edge-of-reach and hence may also appear earlier in a different rollout. This means the target for the state will sometimes have a penalty and sometimes have no penalty. From our investigations, it seems that the resulting contradictory target function or reward signal leads to significant optimization issues. As such there is an extremely delicate trade-off between balancing the penalty to be (1) large enough to address the edge-of-reach issue, but (2) small enough to not cause optimization issues. This highlights the challenge of tackling the edge-of-reach problem! We found our RAVL approach to work much better and more robustly and avoid this optimization difficulty and delicate balancing act.

We would again like to thank you for taking the time to understand and review our work. We especially appreciated your insightful suggestions and encouraging feedback. Please let us know if you have any other comments, concerns, or questions.

We would like to thank the reviewer for their time in reviewing our submission. We are encouraged that you noted we tackled a “surprising problem” in model-based offline RL.

(Question 1) Clarifying out-of-sample and out-of-distribution

We refer the reviewer to Footnote 1 on page 3 of the main paper where we clarify that we use “out-of-sample” and “out-of-distribution” to mean the same thing. We chose to use both terms to highlight connections to past work. IQL [1], for example, contains a very clear explanation of the model-free version of these issues in which they use the term “out-of-sample”, while many other papers (and wider literature in general) use the term “out-of-distribution”.

(Question 2) Generality of our finding

Thank you for raising this question. Theoretically, the edge-of-reach problem may be an issue whenever the effective training dataset (in this case $\mathcal{D}_{\text{rollouts}}$) consists of truncated trajectories that are unable to cover or “reach” the full state space. This is almost certainly the case with the MBPO procedure used in many prominent offline model-based methods (MOPO [2], MOReL [3], MOBILE [4]), combined with limited size offline datasets. Empirically, Table 1 and Figure 5 show that the edge-of-reach problem does indeed occur on the main offline RL benchmark (D4RL), and our new results on V-D4RL (see part B of our general response) indicate wider prevalence of the edge-of-reach problem even for latent-space model-based methods. We give a 1-page summary in Appendix A which concisely explains the conditions which induce the edge-of-reach problem.

(Question 3) Clarification of SAC in Figure 2

We believe there may be a misunderstanding of the meaning of “SAC” in Figure 2. This figure is designed to illustrate that the MBPO procedure [4] (which is the base algorithm in most prominent offline model-based methods) causes the edge-of-reach problem. The MPBO procedure can be described as:

1. generate a dataset of truncated rollouts according to the agent’s policy, and then

2. train the agent (most papers use SAC) on this dataset of rollouts ($\mathcal{D}_{\text{rollouts}}$), and then repeat.

We therefore simply use “SAC” to refer to the base MBPO+SAC procedure (used in MOPO, MOReL, MOBILE…) without our RAVL modification. Please see Table 8 for a summary of the setups. Your question has also prompted us to add Algorithm 1, so thank you for this valuable feedback.

(Question 4) How does RAVL distinguish between states that are edge-of-reach or not?

Thank you for this question. We have included a detailed step-by-step walk-through of the intuition in Part A of our general response. Please let us know whether it is still unclear. We would be happy to provide further explanation.

We once again thank you for your valuable feedback and queries which have helped us to improve the paper. We hope our responses were able to clarify your questions. We would be glad to answer any further questions during the discussion period, so please do not hesitate to let us know if you have remaining concerns or if anything is still unclear. If we have been able to address your concerns, we would humbly ask if you would consider raising your score.

[1] Offline Reinforcement Learning with Implicit Q-Learning. Ilya Kostrikov, Ashvin Nair, Sergey Levine. ICLR 2021.

[2] MOPO: Model-based Offline Policy Optimization. Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, Tengyu Ma. NeurIPS, 2020.

[3] MOReL: Model-Based Offline Reinforcement Learning. Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims. NeurIPS, 2020.

[4] Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning. Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, Yang Yu. ICML, 2023.

Dear Reviewer 1Cj8,

Thank you again for taking the time to review our paper. As the author-reviewer discussion period is about to end (22nd Nov AOE, approximately two days time), we were wondering whether you could let us know if our clarifications were able to answer your questions and clear up any misunderstandings about our algorithm or methodology.

Please let us know if you have any remaining queries or concerns. We hope that we have addressed your questions, and if so, would appreciate it if you would consider revising your score.

Best wishes,

The authors.

(Continued from above.)

B - Additional experimental evaluation

Experiments on Offline Pixel-Based Benchmark (V-D4RL)

We have run additional initial experiments on the challenging offline pixel-based V-D4RL benchmark [1], and are pleased to be able to report that adding RAVL’s edge-of-reach detection and penalization method results in a significant improvement over the baseline on several environments (see Table below).

We observe that RAVL gives a strong boost in performance (+22% overall) on the medium and medexp level datasets, while not helping in the more diverse random and mixed datasets. This observation fits with our intuition of the edge-of-reach problem: the medium and medexp level datasets are likely to have less coverage of the state space and thus we would expect them to suffer more from edge-of-reach issues and the “edge-of-reach seeking behavior” we demonstrated in Section 5.2 of the paper. We note also that these results are without the ensemble diversity regularizer from EDAC [2] (which RAVL uses on the D4RL benchmark), which we anticipate may further increase performance.

Our result on the V-D4RL is particularly notable as the base algorithm used in this pixel-based setting is DreamerV2 [3] (which generates model rollouts in an imagined latent space) rather than MBPO (which uses rollouts in the original state-action space). This suggests that the edge-of-reach problem is more widespread than we had previously indicated.

[1] Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations. Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, Yee Whye Teh. TMLR, 2023.

[2] Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song. NeurIPS, 2021.

[3] Mastering Atari with Discrete World Models. Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba. ICLR, 2021.