We appreciate reviewer px9k for their review and kind comments on our manuscript.

1 The most important finding of this paper, is validated empirically but lacks a theoretical analysis …

As you mentioned in your comment, the two kinds of distributional shifts are rarely distinguished in offline RL papers. This differentiation is fundamental but important for offline RL study. Our work focuses on empirical analysis to reveal this differentiation and proposes a general method to address them separately. We are looking forward to building up the theory by separating these distributional shifts in the near future.

2.The improvement of MCEP upon DQL is limited, which doubts the benefits of MCEP for modern offline RL methods with better designs, such as ReBRAC.

One of the motivations for using MCEP is to reduce the Bellman estimate error brought by OOD actions. Conventional methods use a Gaussian policy that raises many OOD actions under multi-modal distribution. MCEP significantly eliminates these OOD actions and leads to significant performance improvement. DQL utilizes diffusion policies that effectively learn the multi-modal distribution. Hence the Bellman estimate error is reduced. In this case, MCEP can still improve its general performance by optimizing the evaluation policy towards higher-value OOD actions further. we respectfully argue that our contributions do not emphasize strong algorithmic performance, but are two-fold, as concluded below:

1. We propose to distinguish the distributional shifts for Bellman estimate error reduction and the inference-time performance. Existing policy constraints methods (Kumar et al 2019, Nair et al 2020, Wang et al., 2023 etc) focus on addressing the first type but wish to achieve the second type.

2. We propose to address these two types of distributional shifts separately and propose a general algorithm that learns two policies with different policy constraints. Our experiments on both D4RL locomotion and the challenging humanoid illustrate the performance improvement brought by the MCEP.

3. The proposed MCEP is only applicable to policy constraint methods and does not outperform other kinds of sota offline RL methods, such as MCQ and EDAC.

One of the contributions of this work is to distinguish two types of distributional shifts. Our work investigates this phenomenon in policy constraint strengths, hence the resulting method is applied to policy constraints methods, which is a large group of offline RL methods. We are looking forward to investigating this phenomenon in other methods in the future.

Our comparison includes EQL and DQL which are also highly-performed SOTA algorithms. As both the MCQ and EDAC reported performances with task-specific hyper-parameters while we use “similar hyperparameters to locomotion tasks”, the reported performance under different evaluation protocols does not provide a straightforward comparison.

1. What do you mean by 'While the target policy may recover its performance…

Unstable Q learning means that the value of a bad-performed action could sometimes be arbitrarily high but can later back to an accurate estimate when 1) this action is queried during TD learning and 2) the dataset distribution covers this state-action. The target policy might learn this bad-performed action when it’s of arbitrarily high value. But the critic will query this target policy during the TD learning, hence the inaccurate value estimate for this action is fixed. For the evaluation policy, it can learn a different bad-performed action from the target policy’s. But the critic never queries the evaluation policy, hence the value estimate for this action is always arbitrarily high and the behavior of the evaluation policy does not recover to good behavior.

2. How does MCEP 'overcome this drawback' (state-agnostic constraint) discussed in Singh et al.?

Signh et al studies the datasets where the behavior diversity varies among different states. The appropriate constraint strength for a state of low diversity may be too restrictive for the policy to learn behavoir of another state of high diversity. A milder constraint strength enables the policy to learn diverse behavior but increase the Bellman estimate error on the state of low diversity. The MCEP uses the target policy to obtain stable Bellman estimate and the evaluation policy to learn more diverse behavior.

3. I recommend also mentioning Czarnecki et al. in Section 5.1

We added these lines in Section 5.1 “This approach of utilizing the value function of the imperfect teacher policy is originally suggested by Czarnecki et al. (2019)”. Thank you for your thoughtful suggestion.

Typos

We fixed all the typos in the updated manuscript.

Please let us know whether our answers address all your concerns. We are more than happy to answer more if you have further questions and take suggestions to further improve our work.

Thanks for the prompt reply.

Let me elaborate on my concern in the 'improvement room.' According to my understanding and the author's response, there is an "edge” in distribution avoiding severe Bellman approximate errors, and also an edge avoiding OOD states due to OOD actions during evaluation, which is wider than the former one and allows more performant learned policies. Given that the former edge can be pushed approaching the latter one with powerful policy distribution classes (e.g., beyond isotropic deviations) or better policy constraint formulations, my question is, how much room is still in between them? This determines the improvements that MCEP can give to base offline RL methods. Regretfully, the empirical results of MCEP upon DQL seem to suggest the improvement room is small for modern offline RL methods.

Please correct me if there is any misunderstanding.

We thank the reviewer WVL4 for their review of our manuscript.

We appreciate reviewer WVL4 for investigating the experiment details but we find one overlooked setting of our experiment could guide to some misunderstanding.

The improvement may be attributed to the policy constraint strengths of the original method are not well selected…

We use the setting of “similar hyparameters for all locomotion domains” as mentioned in the paper (line 3, Section 5.3 “For D4RL datasets, similar hyperparameters are used”). This setting is widely used in current offline RL literature (Fujimoto et al., 2020, Kostrikov et al., 2021, Hansen-Estruch et al., 2023, Wang et al., 2023, etc). Table 1 reports the best performance under this setting, both for the baseline and for our methods. With this evaluation protocol, the hyperparameters for original methods are well-selected.

In, fig.4, we investigate multiple strengths for the original methods. It is possible that one constraint strength suits a specific task best but does not derive the best general performance. We have an experiment that compares task-wise best hyper-parameters, which can be found in Section A.3 of the Appendix, where in 7 out of 9 tasks, the MCEP outperforms the original algorithm.

For DQL and DQL-MCEP, there are no remarkable differences on most tasks…

One of the motivations for using MCEP is to reduce the Bellman estimate error brought by OOD actions. Conventional methods use a Gaussian policy that raises many OOD actions under multi-modal distribution (Cai et al., 2022, Wang et al., 2023, Hansen-Estruch et al., 2023). MCEP significantly eliminates these OOD actions by using a target policy of strong constraint, which leads to significant performance improvement.

DQL utilizes diffusion policies that effectively learn the multi-modal distribution. Hence the Bellman estimate error is reduced. In this case, MCEP can still improve its general performance by optimizing the evaluation policy towards higher-value OOD actions further.

Based on the above analysis, we can find whether...

The performance of MCEP depends on the stable value estimate, which is related to the hyperparameter of the base algorithm. In our experiments, we use paper-recommended hyperparameters for the target policy.

In Section A.3 (fig. 9), under the recommended hyperparameter (same value for all tasks) for the target policy, only tuning the hyperparameter for the MCEP enables it to outperform the base algorithm that uses task-specific best hyperparameter in 7/9 tasks. This means that it is not necessary to have the best hyperparameter of the base algorithm for the MCEP to achieve better performance.

Although the authors give an ablation study in sec. 5.4, the improvement of MCEP is not very significant and the results seem to be inconsistent with previous figures and tables (See questions).

Under the setting of "similar hyperparameters for all locomotion tasks", all experiment results stay consistent. The improvement of MCEP is significant as under a widely used evaluation protocol, it empowers conventional algorithms to achieve near SOTA performance and provide a further improvement for the SOTA method.

The authors only test their methods on mujoco tasks…

As this is asked by reviewer r8JF, VtJK, WVL4 (3/4 of the reviewers), we answer this question in the general review that can be found at the top of this page.

Which level are you using in fig. 7? …

We apologize for the confusion. Fig. 7 presents the average performance of all versions of datasets for each domain. E.g. the bars above the “halfcheetah” presents the average scores of the “medium”, “medium-replay” and “medium-expert” datasets. With this setting, it is reasonable for Fig.7 and Table 1 to have different scores. We add this explanation in Section 5.4 “The scores for different datasets are grouped for each domain.”

Please let us know whether our answers address all your concerns. We are more than happy to answer more if you have further questions and take suggestions to further improve our work.

References

Fujimoto, S. and Gu, S.S., 2021. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34, pp.20132-20145.

Kostrikov, I., Nair, A. and Levine, S., 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.

Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J.G. and Levine, S., 2023. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573.

Wang, Z., Hunt, J.J. and Zhou, M., 2022. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193.

Cai, Y., et al, 2022, November. TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets. In 2022 IEEE International Conference on Data Mining (ICDM) (pp. 21-30). IEEE.

We thank the reviewer VtJK for their thorough feedback on our work.

(major) Limited Evaluation …

About the evaluation on antmaze, kitchen and adriot, as this is asked by reviewers r8JF, VtJK, WVL4 (3/4 of the reviewers), we answer this question in the general review that can be found at the top of this page.

(major) It turns out that the hyperparameter selection…

An explanation for this phenomenon: Consistent with the results of section 5.4 (Figure 8), a strict constraint hinders the policy from improving beyond the datasets so may show poor performance. With milder constraints, the policy is able to utilize the estimated q function and take some OOD actions of high value during the inference so it achieves better performance over the dataset.

Suggestions about the hyperparameter selection: Based on the results of Section 5.4, we suggest using the recommended constraint strength from the base algorithm paper for the target policy. This ensures a stable q estimate. Then tune the evaluation policy constraint strength to wilder values. Our experiment also follows this tuning protocol for all MCEP methods.

(minor) inconsistent abbreviation

we unify them by using TD3BC (including the figures) in the newly submitted manuscript. Thanks for your thoughtful suggestion.

can your methods be applied to … like CQL?

One of the motivations for using MCEP is to avoid the Bellman estimate error brought by the actor with milder constraints. In the pessimism methods such as CQL, the Q estimate error is reduced by pushing the Q values down for OOD actions. The policy in CQL does not need the policy constraint therefore milder constraint (or non-constraint) policy does not exacerbate the Bellman estimate error and thus the MCEP does not help in these methods.

Please let us know whether our answers address all your concerns. We are more than happy to answer more if you have further questions and to take suggestions to improve our work.

Thanks for your thoughtful discussion.

Can you provide detailed numerical results?

We provide the results of two tasks from Adroit below. The results were taken from the last step of the training, with the average return and standard error from 5 random seeds.

Their hypepameters are: IQL $\tau=0.7, \lambda=2.0$ (tuned among $[0.6, 0.7, 0.8, 0.9] \times [1.0, 2.0, 3.0]$, The AWAC uses $\lambda=1.0$ (tuned among $[0.1, 0.2, 0.3, 0.5, 0.8, 1.0, 1.2]$) and Td3BC uses $\alpha=0.1$ (tuned among $[0.01, 0.1, 0.2, 0.5]$). AWAC-MCEP $\tilde{\lambda}=10.0, \lambda^E = 1000$, TD3BC-MCEP $\tilde{\alpha}=0.1, \alpha^E=0.1$.

TD3BC under this high BC coefficiency is the only method that outperforms behavior cloning. However, the critic learning presents exponentially high values during the training of AWAC and TD3BC. In these cases, the MCEP fails to utilize the arbitrarily high values provided by the critics, resulting in worse performance.

So, MCEP can only aid policy constraint methods? Can it benefit IQL as well?

The MCEP is desgined overcome an overlooked drawback of policy constraints methods, hence it aims to aid policy constraints methods. This weakness is that most policy constraints methods do not distinguish 1) avoiding OOD actions to address unstable value learning and 2) obtaining inference-time performance. Instead, Policy constraints methods treat 1) and 2) as one problem and expect one identical policy constraint to solve both. This is the source of the wellknown tradeoff (Fujimoto et al 2018, Kumar et al 2019). The general idea behihd MCEP is to address 1) and 2) separately, hence a simple two-constraint framework is devised.

IQL is also an instance that follows this general idea and implicitly has two policy constraints. Its value learning can be transformed to actor-critic with implicit target policy (Hansen-Estruch et al., 2023). Its policy extraction derives an evaluation policy that potentially has a different constraint from the target policy.

Pessimistic value methods such as CQL also have the same drawback by solving 1) and 2) with the same degree of conservative. How to address this drawback for pessimism methods is also an open problem (Hong et al 2022, Hong et al 2023).

References

Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J.G. and Levine, S., 2023. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573.

Hong, J., Kumar, A. and Levine, S., 2022. Confidence-conditioned value functions for offline reinforcement learning. arXiv preprint arXiv:2212.04607.

Hong, Z.W., Kumar, A., Karnik, S., Bhandwaldar, A., Srivastava, A., Pajarinen, J., Laroche, R., Gupta, A. and Agrawal, P., 2023. Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets. arXiv preprint arXiv:2310.04413.

We appreciate reviewer r8JF for their review of our manuscript.

For your concern of contribution is limited, we respectfully argue that our contributions do not emphasize strong algorithmic performance, but are two-fold, as concluded below:

1. We propose to distinguish the distributional shifts for Bellman estimate error reduction and the inference-time performance. Existing policy constraints methods (Kumar et al 2019, Nair et al 2020, Wang et al., 2023) focus on addressing the first type but wish to achieve the second type.

2. We propose to address these two types of distributional shifts separately and propose a general algorithm that learns two policies with different policy constraints. Our experiments on both D4RL locomotion and the challenging humanoid illustrate that the MCEP empowers conventional methods (TD3BC and AWAC) to achieve near SOTA performance and shows a consistent performance improvement on SOTA algorithm DQL.

For example, given that $\pi_e$ is not involved …

We provide a new experiment in Appendix Section A.4 to illustrate the difference between these two design options. Our algorithm optimizes two policies together to allow parallelizing their learning thus the training time is the same as the base algorithms. This is powerful for algorithms that require long training time for the policy (e.g. DQL which utilizes diffusion policies).

Have you tried to train a greedy policy …

While this idea works for the grid example, a greedy policy in larger MDPs will inquire the q function about OOD actions where the estimated Q is arbitrarily high. Based on this, the evaluation policy still needs a policy constraint (though milder). Pessimism Q methods (CQL etc) do not have this issue so their policy does not need the policy constraints.

Why have’nt you tested other environments …

As this is asked by reviewers r8JF, VtJK, WVL4 (3/4 of the reviewers), we answer this question in the general review that can be found at the top of this page.

Figures are not readable when printed out.

We apologize for the inconvenience during your review. We changed the font size in the new manuscript (as well as fixing the typos). Again, we appreciate your effort and suggestion.

Please let us know whether our answers address all your concerns. We are more than happy to answer more if you have further questions and to take more suggestions to improve our manuscript.

References:

Kumar, A., Fu, J., Soh, M., Tucker, G. and Levine, S., 2019. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32.

Nair, A., Gupta, A., Dalal, M. and Levine, S., 2020. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359.

Wang, Z., Hunt, J.J. and Zhou, M., 2022. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193.

To better resolve a common concern raised from reviewers, we conduct an experiment on a diverse set of 16 robotic manipulation tasks on the KUKA’s IIWA robot, from the CompoSuite (Jorge et al 2022, Hussing et al 2023). Below we show the average success rates among 100 episodes of evaluating the policy from the last step of a 1-million step training. Results are averaged among 5 random seeds.

Averaged Success Rates for 16 Robotic Manipulation tasks

Medium

Medium-Replay

We consider similar hypermeter for all tasks. We tune CRR $\lambda = [0.4, 0.6, 0.8, 1.0, 1.2]$ and uses softmax inference-time policy improvement (see discussion with reviewer r8JF). For TD3BC $\alpha = [1.0, 2.0, 3.0, 4.0]$. IQL uses $\tau=0.7, \lambda=3.0$ as recommended in (Hussing et al). For TD3BC-MCEP, $\tilde{\alpha}=2, \alpha^E = [4.0, 6.0, 8.0, 10.0]$.

In this experiment, we observe that the MCEP is able to provide consistent and significant performance improvement in complex tasks. These results should resolve the concern of whether the MCEP works in complex domains.

References

Jorge A. et al., CompoSuite: A compositional reinforcement learning benchmark. In 1st Conference on Lifelong Learning Agents, 2022a.

Hussing, et al., 2023. Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning. arXiv preprint arXiv:2307.07091.