BDQL: Offline RL via Behavior Diffusion Q-learning without Policy Constraint

We express our gratitude for your comprehensive feedback. We've made careful considerations and addressed each of your concerns.

Q1: The good property of two-stage training in BDQL is only validated in MuJoCo tasks, which are relatively simple. There is a lack of experiments on more complex datasets, such as AntMaze and Adroit domains from MuJoCo, or even heteroskedastic datasets [2]. The performance of BDQL-SWA on MuJoCo tasks still trails behind modern offline RL methods.

A1: Although the results of BDQL are not outstanding, the experiments is achieved under the condition of "without constraint to overcome OOD". Combined with our newly added theoretical analysis, we comprehensively proves that BDQL is an algorithm capable of solving offline RL without the need for constraints.

Q2: Also, it is only validated empirically, without theoretical analysis.

A2: We have added some theory to better analyze the two-stage property of BDQL. In Conclusion 1 of the new version of the paper, we find that the optimality of BDQL is based on the small enough distance (Total Varational distance between the estimated behavioral policy and the true behavior policy). Building on this conclusion, we divided the training stage based on the distance between the current training policy and the true behavior policy. We observed that the experiment perfectly matched this analysis.

Q3: It is not the first time diffusion models have been utilized to model complex behavior distributions, which has been done by Chen et al. [1]. The authors should explicitly clarify this and sufficiently discuss the difference with them.

A3: There is no apparent peculiarity in the way we use diffusion model. The crucial core is the unique role of diffusion policy. In the new version of the paper, we demonstrate through theoretical analysis (refer to Conclusion 1 and the following description) that the theoretical framework of BDQL highly depends on the precise behavioral modeling provided by diffusion model.

Q4: Why 'The output of the diffusion policy is the deterministic action rather than the distribution of action'? If I understand correctly, the diffusion policy models a stochastic state-conditioned action distribution as Equation (3) rather than a single deterministic action. So Equation (8) should be $\mathbb{E}*{s_t \sim \mathcal{D}, a \sim \pi*\theta(s_t)} [Q_\phi(s_t, a)]$? Does this one still follow the deterministic policy gradient (DPG) theorem?

A4: From the details of the diffusion model, this action is sampled from a distribution. However, from an external perspective, the diffusion model takes in a state and outputs an action. This aligns with the definition of a deterministic policy and can be updated using DPG. Similar approaches are also adopted by Diffusion Q-learning [1].

[1] Wang, Z., Hunt, J. J., & Zhou, M. (2022). Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193.

Q5: The name of 'theoretical performance' is inappropriate. Better names can be 'online validation performance,' 'best performance,' or 'ideal performance.'

A5: We have used "best result" in our new paper.

Should there be any misinterpretation on our part regarding your inquiries or if we have not sufficiently resolved your concerns, please bring it to our attention at your earliest convenience. We look forward to any additional advice or guidance you can provide.

Your detailed comments are greatly appreciated. We have carefully addressed each point, and your insights have contributed significantly to improving our work.

Q1: First, from the experimental studies, the results of BDQL is not convincing. The performance of BDQL is close to other competitors and BDQL-SWA is even inferior to other competitors.

Q2: From the authors' explanation, the Theoretical performance refers to the on-policy scenario, and the Practical performance refers to the offline scenarios. In offline scenario, the performance of BDQL-SWA is still not good enough.

A1&A2: We address these two questions together. BDQL and its variants achieve results comparable to the strongest baseline (ensemble-based methods). While this may not be a breakthrough, it is important to note that this result is achieved without introducing constraints and with OOD preserved. This experiments, along with our newly added theoretical insights, demonstrate that offline RL may not necessarily require constraints to solve.

Q3: In SWA, the author mentions ``SWA averages the multiple checkpoints during the optimization''. Do you mean averaging the parameters of the models at different training time? So it is similar to the Target Actor/Critic network trick in most offline RL methods?

A3: Indeed, SWA averages different checkpoints during training, but it differs from the Target Actor/Critic network trick. The key distinction is that the Target Actor/Critic network actively participates in training, whereas SWA does not. SWA averages different checkpoints during training to stabilize fluctuations and avoid collapse.

Q4: Some offline-RL baselines, such as BEAR are missed in experiment, and the recent popular SPOT [1] method is not considered in experiments as well. It is suggested to consider the baselines in offline-RL methods

A4: The descriptions of the support constraint value function in BEAR and SPOT inspired our proof, and we have cited both of these papers. The instability of BEAR's performance is a major reason why we did not consider it in our baseline. The strength of the SPOT algorithm lies in its offline-to-online fine-tuning while the performance is comparable to TD3+BC in addressing offline RL (SPOT: 773/9 = 85.89, TD3+BC: 84.83). Meanwhile, our baseline also includes ensemble-based methods, which are among the strongest algorithms.

Q5: The author claims that the BDQL has sufficiently solves the offline RL problem (OOD issue), and the OOD issue only causes fluctuation in training. The authors tries to illustrate this point with some ablation studies. However, this claim seems weak. It is suggested to add some theoretical analysis supporting this claim.

A5: In the new version of the paper, we analyze the optimality of BDQL and its tenable conditions (please see Section 4). We find that when the estimated behavioral policy is close enough to the true behavioral policy, BDQL can converge to the optimal policy. Inspired by this conclusion, we divide the training process into two stages based on the distance between the current training policy and the true behavior policy. This leads to improved analytical results and better demonstrationl in our experiments (Figure 4).

Q6: Actually, the diffusion model is time-costly in model inference. Will this issue also occur in BDQL? Some ablation studies on computation issues are suggested.

A6: we compare the training time between BDQL and BQL. The training time of BDQL (1492 min) is approximately 2.5 times that of BQL (608 min). In future work, we will consider further optimizations in terms of algorithm and implementation to reduce the time overhead.

In case our understanding of your questions is not accurate or if your concerns remain unaddressed, we encourage you to inform us promptly. We are keenly awaiting any additional suggestions or guidance you might offer.

We would like to know if our response has addressed your concerns. Additionally, we are eager to hear your feedback on the newly added proof section. We look forward to your response and any further suggestions.

Thank you sincerely for your thorough and insightful comments. We have taken each question seriously and provided detailed responses.

Q1: The significance of section 2.2 is unclear, since both methods are not used in the current paper.

A1: In Section 2.2, we introduce on-policy and off-policy learning. The former is highly relevant to our algorithm since BDQL can be viewed as a form of on-policy learning in the context of offline RL. When introducing off-policy learning, we also highlight the challenges in offline RL. This section serves as essential groundwork for the subsequent discussions.

Q2: The SARSA update requires the assumption that the offline data is coming from trajectories, and the data collecting policy is a single stationary policy.

A2: SARSA can also be updated from transitions, not only trajectories. Offline datasets can be assumed to be collected by the behavior policy, which allows update by SARSA. This assumption has been adopted by on-policy style offline algorithms like Onestep RL [1] and BPPO [2].

[1] Brandfonbrener, D., Whitney, W., Ranganath, R., & Bruna, J. (2021). Offline rl without off-policy evaluation. Advances in neural information processing systems, 34, 4933-4946.

[2] Zhuang, Z., Kun, L. E. I., Liu, J., Wang, D., & Guo, Y. (2022, September). Behavior Proximal Policy Optimization. In The Eleventh International Conference on Learning Representations.

Q3: The choice of using SARSA to train the critic is actually confusing. According to the proposed algorithm, in the statistically asymptotical case, optimization is done perfectly, and offline data has good coverage, the critic will converge to the Q-function of the behavior policy, which might not be a strong policy, and the diffusion policy is just the argmax policy according to the Q-function of the behavior policy, which might be better than the behavior policy, but the performance is still not guaranteed. So it is hard to see why even in the most ideal setting this algorithm would return a strong policy.

A3: One of our innovations is updating the policy using only the critic without introducing constraints. Whether the learned policy is optimal or not can be addressed by our added proofs. In essence, when the policy estimated by BC is sufficiently close to the true behavior policy, the optimality of the policy is guaranteed.

Q4: The argument on regularization for ood from the current algorithm is not very convincing. It seems like an alternative way of regularizing with behavior cloning with a diminishing regularization coefficient. Using this perspective, one can also suspect if this varying objective is causing the instability in the practical performance.

A4: We may not fully understand this question and are unsure if the reviewer is asking if SWA is an alternative OOD regularization method. If so, our response is as follows; SWA does not participate in training and does not alter the original optimization objectives. This fundamentally distinguishes the role of SWA from other constraints. The training of the original BDQL is unstable, and SWA is used to eliminate this instability

Q5: No concrete algorithm box is provided.

A5: We have added the algorithm box in appendix and the Figure 1 can also clearly express the algorithm logic.

Q6: The usage of the term "theoretical performance" in the experiment section is confusing.

A6: We have replaced the “theoretical performance” with “best result”, representing the best result obtained from online evaluation.

If we have misunderstood your queries or failed to address your concerns adequately, please do let us know promptly. We eagerly anticipate any further suggestions or guidance you may provide.