Is Value Learning Really the Main Bottleneck in Offline RL?

Thank you for the detailed review and constructive feedback about this work. We especially appreciate the reviewer's feedback about statistical significance and the clarity of our claims. Following the reviewer’s suggestion, we have added more seeds and variance metrics to improve the statistical significance of the results. We believe these changes have strengthened the paper substantially. Please find our detailed responses below.

• “It is a bit concerning that the main results in Figures 1, 2, 6 are over only 4 seeds.” / “the same figures do not report any information on variance of performance”

Thanks for raising this point. Following the suggestion, we’ve added 4 more seeds to Figures 1, 2, and 6 by adding 7744 more runs (now we have 8 seeds in total for every experiment in the paper) and have reported standard deviation metrics in our data-scaling matrices. Please find the improved results in Figure 1 in the additional 1-page PDF. Our conclusions remain the same, and we have updated these results in the current version of the paper.

• About extreme actions

We fully agree with the reviewer that extreme actions do not necessarily mean they are more optimal. Our original intention was to show that AWR has limited expressivity compared to DDPG+BC, because AWR actions always lie in the convex hull of the dataset actions (at least in principle), while DDPG+BC can go outside of it. This resulted in more extreme actions for DDPG+BC (Figure 3), but this is not to say that extreme actions are better. We apologize for the confusion and we will clarify this point in the draft to prevent such potential confusion.

• “the claim to “always use” (above Section 5) is too strong”

Thanks for the suggestion. We initially used the term “always” because we found that DDPG+BC is better than or as good as AWR in most of the cases (15 out of 16 settings — please refer to Table 1 in the new 1-page PDF). However, following the suggestion, we have toned down the takeaway by removing the word “always” in the current draft.

• “DDPG+BC’s focus on limited exploration” / “by encouraging a policy to be close to the behavior policy, the test-time generalization should naturally be better since it will not deviate much from those states”

Thanks for the great question. We also believe that, in general, the higher the BC coefficient in DDPG+BC is, the better it can prevent encountering out-of-distribution states at test time. (We’d also like to note that this also applies to AWR since it has a temperature hyperparameter of a similar role.) However, in practice, due to function approximation or imperfect policy learning, the agent would almost always visit out-of-distribution states, even when there’s only the BC term, and thus the generalizability of the policy can still significantly affect performance even in this case. We provide one example of such evidence in Appendix C. That said, as the reviewer mentioned, we expect that a small BC coefficient would incur even further challenges in generalization.

• About the Fujimoto et al. paper

Thanks for the pointer! We think this paper indeed points out similar limitations of the Bellman error. We will cite and discuss this paper.

We would like to thank the reviewer again for raising important points about statistical significance and clarity, and please let us know if there are any additional concerns or questions.

Thank you for the detailed review and constructive feedback about this work. It appears that we were not entirely clear about our main messages in the initial draft, which we believe may have caused some confusion about our claims. Below we describe how we have revised our paper to prevent potential confusion and have added new results about aggregation metrics and generalization analysis.

• The results are not as conclusive as the paper suggests; they indicate that sometimes policy learning is the bottleneck and sometimes value learning is the bottleneck.

Thank you for raising this valid point. We believe our initial draft was not entirely clear about our analysis framework. We first would like to clarify that there are two types of information we can obtain from the data-scaling matrices (Figure 1).

• (1) By looking at each individual matrix, we can see scaling behaviors with the amounts of value and policy data in that specific setting.
• (2) By comparing different matrices from different value/policy algorithms, we can understand the effect of each algorithmic component.

As the reviewer pointed out, if we look at individual matrices (the first perspective), some matrices are value-bounded (i.e., the performances are more affected by the amount of value data) and others are policy-bounded, and the results appear to be less clear and problem-dependent. However, if we compare different algorithms (the second perspective), we can observe a much clearer trend in our results: namely, (1) the choice of a policy extraction algorithm often affects performance more than the choice of a value learning algorithm (except antmaze-large), and (2) DDPG+BC is almost always better than AWR. Please refer to our global response for aggregation metrics that quantitatively support these observations.

We then find the reason behind this difference between AWR and DDPG+BC by now looking at individual matrices (Section 4.3). The gc-antmaze result in Figure 2 shows a clear difference between the two algorithms: with AWR, an increase in value data doesn’t necessarily translate to performance improvement, unlike with DDPG+BC. This suggests that AWR, one of the most widely used policy extraction algorithms, imposes a “bottleneck” that inhibits the full use of the learned value function (the “Takeaway” box in Section 4). In this sense, we argue that policy extraction is often the main bottleneck in current offline RL algorithms (esp. any method that involves weighted/filtered BC, including the recent scaling work [4]). Please note that this is different from saying that every individual data-scaling matrix is policy-bounded (which is indeed not true).

We believe our initial draft was not very clear about these points, and will revise the manuscript to clearly convey these findings.

• “the results are pretty hard to parse” / “Lack of clear aggregation metrics”

Thank you for the helpful suggestion. Following the suggestion, we have added two types of quantitative aggregation metrics. Please refer to our global response for the details.

• “It is not clear why there is such focus on the generalization of the policies and not the values.”

Thanks for raising this question. First, we would like to note that our main focus in the second analysis (Section 5) is not on comparing value generalization and policy generalization (unlike our first analysis); our main point is rather to show that test-time generalization (regardless of policy or value) is one of the important yet overlooked issues in offline RL. We used the phrase “policy generalization” in the paper because the policy is what is deployed at test time, but we didn’t intend to mean that policy generalization is necessarily more important than value generalization. Instead, we wanted to show how current offline RL algorithms are often already great at learning near-optimal actions on in-distribution states, and how bad they can be on test-time out-of-distribution states. This observation highlights an important (but often overlooked) open question in offline RL, which is very different from the previous main focus on "value learning" about pessimism and conservatism. We apologize for this potential confusion (which we think stems from some phrases in Sections 1, 3, and 5), and will revise the paper to make this point very clear (e.g., we will use just “generalization” not “policy generalization” whenever it’s more appropriate).

That said, we do have some empirical results that allow us to compare policy and value generalization, and we discuss these aspects in the global response above in case it’s of interest to the reviewer.

• The use of IQL+AWR in our offline-to-online RL experiment / MSE under $d^{\pi^*}$

Thanks for the suggestion. Following the suggestion, we have repeated the same experiment with IQL+DDPG+BC and additionally measured the MSE under $d^{\pi^*}$. We obtained similar results and our conclusion remains the same. Please refer to our global response for the details.

• “In general, the paper tries to cram in too many semi-related results.”

Our primary motivation in this work is to understand the best way to improve the performance of current offline RL. To this end, we believe conducting a holistic analysis of different elements that affect the performance of offline RL is necessary, and thus we carried out such analyses (across value learning, policy extraction, and generalization) in this paper. That being said, to provide more coherent insights, we have substantially revised several subsections and paragraphs in the current version, which we hope helps address this concern.

We would like to thank the reviewer again for raising important points about our claims, which substantially helped strengthen the paper. Please let us know if we have addressed the reviewer’s concerns and if so, we would be grateful if you are willing to upgrade the score.

Thank you for the detailed review and constructive feedback about this work. We especially appreciate the reviewer's feedback about statistical significance as well as the question on our claim about value learning. Following the reviewer’s suggestion, we have added variance metrics as well as four more seeds (8 seeds in total) to improve the statistical significance of our results. We believe these changes have strengthened the paper substantially. Please find our detailed answers below.

• “lack of variance measures”

Thanks for raising this point. Following the suggestion, we have added standard deviations to our data-scaling matrices, and report them in Figure 1 in the additional 1-page PDF. We have also updated our paper accordingly.

• “For some experiments in empirical analysis 1, an increase in data leads to a decrease in performance.”

As the reviewer mentioned, there are some cases where an increase in data doesn’t necessarily lead to improved performance. We believe this is mainly due to statistical errors. To address this concern, (1) we’ve added 4 more seeds to our data-scaling matrices by adding 7744 more runs (now we have 8 seeds in total for every experiment in the paper), and (2) we have added variance metrics to the table. Please find them in Figure 1 in the new 1-page PDF. There are still some rare cases where an increase in data doesn't lead to performance improvement, but the new standard deviation metrics tell us that this is likely because of statistical noise. We believe that these changes have significantly strengthened the statistical significance of our results.

• About the title / overestimation in value learning

We agree that overestimation is an important issue in offline value learning in general, and would like to clarify the scope of our claims. Concretely, we would like to clarify that we’re not suggesting that value learning is less or not important compared to policy learning in general. Rather, our claim is that, (due to recent advancements in offline value function training) current state-of-the-art offline RL methods often tend to be at the point where improvements in policy extraction and generalization can lead to larger improvements in performance; in this sense, we claim that policy extraction and generalization are often the main “bottlenecks” in current offline RL, hence the title. As we show in our experiments (Figures 1 and 2), we are at the point where improvements in value functions (by adding more data) often do not translate to policy performance if the policy learning algorithm is not chosen appropriately. Hence, we think it is useful to concretely highlight and emphasize policy learning as a bottleneck at this point since the community has focused on value learning substantially. Thanks for asking this question, and we have revised our paper (Intro and Section 3) to make this point clearer.

• “For empirical analysis 2, I am wondering if the online data mainly improves value learning, policy learning, or both of them.”

Thanks for the interesting question. In this paper, we mainly measure the accuracy of the policy because the policy is what is deployed at test time. We expect additional online data would improve both the value and the policy in general, but it might be a bit challenging to faithfully measure the accuracy of the learned value function (as opposed to policy accuracy), given that (1) the Bellman loss often does not necessarily correlate with the actual performance [3] and (2) offline value functions (especially with techniques for mitigating overestimation like IQL/CQL) often involve pessimism/conservatism, which makes it difficult to directly compare against the actual ground-truth Q value. While we do have a comparison between the value and the policy in terms of data-scaling in the paper, we leave dissecting the effect of each component for generalization (with additional online data) for future work.

We would like to thank the reviewer again for raising important points about statistical significance and the score of our claims, and please let us know if there are any additional concerns or questions.

We would like to thank the reviewer for appreciating our changes. We believe these updates and clarifications have indeed strengthened the paper.

Thank you for the positive review and constructive feedback about this work! We especially appreciate the question on discrete-action MDPs. Please find our answer to the question below.

• “... most of the empirical results are in environments with continuous action spaces. Do you expect the results to carry over to environments with discrete action spaces?”

Thanks for raising this point. There are two main differences between discrete-action and continuous-action MDPs: (1) discrete-action MDPs do not always require a separate policy extraction procedure, as we can directly enumerate over actions to choose the argmax action, and (2) DDPG is not straightforwardly defined in discrete-action MDPs. Hence, one of our takeaway messages (“use DDPG+BC instead of AWR”) may not be directly applicable to discrete-action MDPs. However, we expect that our main findings — namely (1) policy extraction can inhibit the full use of the learned value function (if there’s a separate policy extraction step) and (2) test-time policy generalization is one of the significant bottlenecks in offline RL — still apply to discrete-action MDPs as well. For example, recent works in RL fine-tuning of LLMs, which use a discrete action space, have observed similar findings to us: Tajwar et al. (2024) [1] showed that sampling actions (i.e., responses) from the policy (akin to sampling actions from the policy for making an update in DDPG+BC) lead to better performance than AWR that does not sample a new action from the policy for training, and Cobbe et al. [2] showed the effectiveness of test-time verifier reranking methods, which is closely related to the test-time generalization argument and TTT/OPEX in our paper. While of course we do not focus on LLMs in this study, these prior works illustrate that similar findings are applicable to discrete actions. We will add a discussion about discrete-action MDPs in the paper.

We would like to thank the reviewer again for the helpful feedback and please let us know if there are any additional concerns or questions.