Adjustable Quantile-Guided Diffusion Policy for Diverse Behavior Generation in Offline RL

Thanks for your feedback, and we would like to provide the following clarifications.

Q: (Unclear empirical benefit) Whilst the paper shows a strong improvement on speed, the empirical benefit of the approach is unclear, as there is only a slight gain over the baseline DQL. This is compounded by the fact that the main empirical evaluation in Section 6.1 maximizes over an extremely large 70 different hyperparameter configurations, which is extremely unrealistic for offline reinforcement learning. Coupled with the sweep in Figure 4, the main empirical evaluation is likely maximizing over statistical noise. The results over a smaller hyperparameter space in Section 6.3 do not show a clear improvement.

A: Thank you for your suggestions. IDQL's experimental results are quoted from the result of IDQL-A. "-A" is refered as reported results that allow any amount of tuning and "-1" as results that only allow one hyperparameter to be tuned between domains[1]. And IDQL-1's results are only 1150.9. So, the performance of qGDP, 1220.6, in a smaller hyperparameter space is sufficiently good.

[1] Hansen-Estruch P, Kostrikov I, Janner M, et al. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023.

Q: (Quantitative metric of success in toy experiments) Whilst the authors helpfully compare the effect of different guidance and sampling schemes on their toy dataset in Figure 2-3, it is unclear what the optimal or desired behavior is. It would be helpful to include some indication of what the optimal behavior should be and some kind of metric to assess this (e.g. conforming to some desired distribution).

A: Thank you for your suggestions. The results in Fig. 2 are generated based on the toy datasets, as illustrated in Fig. 1. This experiment is primarily conducted for a qualitative analysis of the impact of hyperparameters on the results produced by different methods. The goal is to assess whether these methods exhibit generalization capabilities and to examine the effects of hyperparameters on the generated results, so there is no specific desired distribution.

Q: (Incomplete speed discussion) Whilst Table 4 shows an improvement over DQL, it would be also valuable to compare runtime to related approaches, IDQL and SfBC. Furthermore, details are missing on which environment exactly is being timed in the table.

A: Thank you for your suggestions.We test the release codes of DQL, IDQL, and SfB with the halfcheetah-medium-v2 task on a single RXT 2080Ti graphics card, and their runtimes are compared below:

method | critic | actor | sampling | total

DQL | 8.6ms$\\times$2e6 | 12.4ms$\\times$2e6 | 51.2s$\\times$40 | 12.9h

IDQL | 3.3ms$\\times$1.5e6 (critic and actor) | 70.0$\\times$6 | 1.5h

SfBC | 68.4ms$\\times$4.9e4 | 39.1ms$\\times$5.9e5 | 50.6s$\\times$20 | 7.6h

qGDP | 9.3ms$\\times$4e5 | 5.9ms$\\times$4e5 | 10.0s$\\times$70 | 2.0h

where, 'critic,' 'actor,' and 'sampling' respectively refer to the time spent on value function training, policy function training, and interacting with the environment of these algorithms. The recorded form of these terms is the time required for a single update of the value function multiplied by the number of updates, the time required for a single update of the policy function multiplied by the number of updates, and the time required for model evaluation (10 interactions with the environment) multiplied by the number of evaluations.

It should be noted that IDQL is implemented based on JAX, resulting in shorter runtimes. In comparison, both qGDP and DQL are implemented in PyTorch. Specifically, in the actor update phase (related to training the diffusion model), qGDP exhibits significantly shorter runtime compared to DQL. During the model evaluation phase, qGDP reduces time consumption through parallelization.

Most importantly, the presented runtimes for DQL, IDQL, and SfBC are for a single run. If hyperparameter tuning is required, they would incur several times more runtime. In contrast, qGDP has completed hyperparameter tuning during the 70 model evaluation runs.

Thanks for your feedback, and we would like to provide the following clarifications.

Q: One of my major concerns is as follows. One main advantage of the proposed method is that it can extrapolate high-value samples outside the training distributions based on the reward function’s trend. However, why is this valid for offline RL settings? The values of out-of-distribution samples are usually penalized as the pessimism of value estimation in order to achieve conservative and best performant policies in online evaluation. Please justify why simple extrapolation based on the trends of reward function is valid. One can easily construct a counterexample to make qGPD totally failed in the bandit example, by setting a first increasing then suddenly decreasing reward structure, e.g., by letting the purple regions in Fig. 3 (c)(f) to have very low reward values.

A: Thank you for your question. In practical applications of offline RL [1][2], it is typically allowed to interact with the environment to a limited extent. In these scenarios, we can utilize these interactions to select appropriate input quantile. In settings where ”a reward structure first increases and then suddenly decreases”, we can also adjust the input quantiles to a suitable value through a small number of interactions to avoid generating samples with low rewards. If applying qGDP to a scenario where online interaction is impractical, we can directly constrain the input quantiles to be less than 1.

[1] Fang, Xing, et al. Offline Reinforcement Learning for Autonomous Driving with Real World Driving Data. IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022.

[2] Singh, Avi & Yu, Albert & Yang, Jonathan & Zhang, Jesse & Kumar, Aviral & Levine, Sergey. COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning. 4th Conference on Robot Learning (CoRL 2020).

Q: Another problem bothers me is the performance improvement over IDQL. From what I understand, the qGDP only improves over IDQL by predicting over more quantiles, so why the performance of qGDP can be better than IDQL if the quantiles considered for two methods are the same, or is the improvement just caused by qGDP searches over a larger space of quantiles or with a smaller grid size? If the improvement mainly comes from this, I cannot be convinced that qGDP is a very novel method.

A: Thank you for your question. qGDP has two main advantages over IDQL: Firstly, qGDP has generalization capabilities and can generate samples with quantiles greater than 1, while IDQL cannot operate when the quantile is greater than 1. As explained in Section 5, the samples generated by qGDP align with the trend of action values (or rewards), and Section 6.2 demonstrated that qGDP performs better in some tasks when the input quantile is greater than 1. Secondly, adjusting the generation preference in IDQL requires retraining the model, whereas qGDP only requires changing the model's input. This significantly reduces the computational cost required for parameter tuning.

Thanks for your feedback, and we would like to provide the following clarifications.

Q: The quantile labels rely on the action values Q learned by IQL, which may have overestimation bias. Errors in Q would propagate to incorrect quantile labels, which could be amplified for higher quantiles. This could negatively impact the quality of the diffusion model training.

A: Thank you for your question. In our code implementation, we have utilized the Clipped Double Q-learning technique provided by TD3[1] to mitigate Overestimation Bias. [1] Fujimoto S, Hoof H, Meger D. Addressing function approximation error in actor-critic methods[C]//International conference on machine learning. PMLR, 2018: 1587-1596.

Q: The quantile input y for guiding the diffusion model is constrained to the range [0,1] or not ? It will limit the scope of behavioral patterns that can be generated. Values greater than 1 may allow more generalization, but this is not explored. And I expected to find the experiment results of different value of y for different generation, but failed. The paper over-emphasizes the impact of tuning the guidance scale, even though adjusting the scale for diversity is intrinsic to diffusion models themselves. However, the main innovation of the paper is using the quantile network to label samples and train the diffusion policy conditioned on quantiles. More analysis is needed on how the quantile inputs specifically affect diversity, rather than just tuning the guidance scale which is a general feature of diffusion models.

A: Thank you for your question. The quantile input can take values larger than 1. In Figure 3, we visualize the generation results of qGQP under different guidance scales and quantile inputs. In each subfigure, points of different colors represent generated samples for quantile inputs y=0.5, 0.7, 1.0, and 1.3. The figure is evident that qGQP exhibits strong generalization capabilities, generating samples not present in the dataset and consistent with the underlying trend. Furthermore, on the D4RL dataset, we experimented with different quantile inputs. The results indicate that both quantile input and guidance scale significantly impact the algorithm's performance, as shown in Figure 4.

Q: While the quantile input is motivated by IDQL, the paper does not sufficiently differentiate the advantages of the quantile mechanism compared to IDQL itself. The paper does not sufficiently differentiate the advantages of the quantile input compared to just using weighted actions like in IDQL. The results between qGDP and IDQL in Table 1 are fairly close. More analysis is needed to clearly explain the relationship between the two methods and highlight the unique benefits of the quantile inputs.

A: Thank you for your suggestions. qGDP has two main advantages over IDQL: Firstly, IDQL increases the probability of the diffusion model generating high-value actions by reweighting samples. So it cannot generate samples with higher values outside the dataset. In contrast, qGDP learns the probability distribution of actions conditioned on different quantiles. Therefore, when the true action value function changes smoothly, qGDP can output action distributions corresponding to higher action values (associated with quantile inputs greater than 1). Secondly, adjusting the generation preference in IDQL requires retraining the model, whereas qGDP only requires changing the model's input. This significantly reduces the computational cost required for parameter tuning. On the locomotion-v2 task set and the tasks of antmaze-umaze-v0 and antmaze-umaze-diverse-v0, our method significantly outperforms IDQL. In the remaining tasks, the reason for qGDP being weaker than IDQL may be attributed to the excessively large state space. The probability of each state being labeled with different labels significantly decreases, leading to insufficient training for quantile-guided diffusion.

Thanks for your feedback, and we would like to provide the following clarifications.

Q: The paper lacks an introductory explanation of quantile networks and their relevance to the proposed method, potentially hindering comprehension for readers less versed in the domain...

A: Thank you for your suggestions. The relevant explanations are provided below, and we will incorporate these details into the main text in the subsequent revisions： The quantile network, proposed by Dabney et al.[1], is designed to describe the action value distribution for each state-action pair (i.e., estimating the values corresponding to different quantiles under this distribution). It constructs the corresponding Distributional Bellman Operator to improve the performance of DQN. IQL employs a similar approach to characterize the action value distribution under the behavioral policy, and uses the values corresponding to high quantile points as an approximate estimate for the optimal action. This helps in avoiding querying out-of-sample actions in the TD loss. In this work, we utilize the quantile network to describe the distribution of action values under the behavioral policy, assign different labels to samples based on the value according to the quantiles, and finally use these labeled samples to train a diffusion model. This trained diffusion model is able to generate appropriate actions based on the input quantiles and guidance scales and exhibits different decision preferences. [1] Dabney W, Rowland M, Bellemare M, et al. Distributional reinforcement learning with quantile regression, Proceedings of the AAAI Conference on Artificial Intelligence. 2018, 32(1).

Q: The core innovation, applying quantile labels to guide the diffusion process, appears somewhat incremental, casting doubt on the method's novelty. Clarification on the specific novelties and contributions of qGDP would be beneficial.

A: Thank you for your suggestions. The main contributions of qGPD are as follows: Hyperparameter tuning is the big problems that still lack a satisfying solution.[2] Most offline RL methods select the optimal hyperparameters by evaluating the performances of models trained with different hyperparameters by interacting with the environment[3]. This implies that these methods require repeating the training process much times. Our approach achieves a behavior-controllable policy with just one training iteration. By adjusting the input quantile and guidance scale, this policy can be biased towards generating actions with higher estimated values or towards actions that closely resemble the behavioral policy. [2] Levine S, Kumar A, Tucker G, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems[J]. arXiv preprint arXiv:2005.01643, 2020. [3] Prudencio R F, Maximo M R O A, Colombini E L. A survey on offline reinforcement learning: Taxonomy, review, and open problems[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023.

Q: The introduction does not adequately articulate how qGDP surmounts the limitations of prior diffusion-based offline RL methods. It would be advantageous to outline explicitly how qGDP addresses the deficits of previous diffusion-based methods in the introduction.

A: Thank you for your suggestions. The limitations of prior diffusion-based offline RL methods and the advantages of qGDP are listed below: a. IDQL increases the probability of the diffusion model generating high-value actions by reweighting samples, so it cannot generate samples with higher values outside the dataset. In contrast, qGDP learns the probability distribution of actions conditioned on different quantiles, thus qGDP can output action distributions corresponding to higher action values (associated with quantile inputs greater than 1).
b. DQL uses the Q-network to guide the training of the diffusion model, so it involves computing gradients for a relatively deep neural network. qGDP is designed to train the diffusion model to generate samples at different quantiles, so the computational cost of qGDP is comparable to that of a regular Diffusion model. c. Adjusting the generation preference in other diffusion-based offline RL methods requires retraining the model, whereas qGDP only requires changing the model's input. This significantly reduces the computational cost required for parameter tuning.