We appreciate the insightful comments and suggestions from the reviewer. We will try our best to answer those questions raised by the reviewer to remove his/her concerns.

The novelty concern is replied in the above general response.

We appreciate the reviewer for raising the question regarding why consistency models are able to achieve higher online performances than diffusion models. Our conjecture is that the expressiveness of a model is more essential in offline setting than online setting, especially when multi-modal distributions appear in offline dataset and the algorithm is of the behavior-cloning (BC) type. Our results of BC in offline setting as Tab.1 show that the expressive policies can significantly improve the BC performances, which can be helpful for policies to quickly converge in offline RL. However for online RL in Markov decision process, the theoretically optimal policy can always be deterministic, which indicates that the expressive models will shrink back to unimodal to approach optimality, thus expressiveness may only help with initial exploration instead of final convergence of the policy. This conjecture can be hard to verify consistently due to the heterogeneous distributions of different tasks. However, this is an interesting question and we will explain this in a modified version of the paper.

Statistical significance. For all the results derived by ourselves, we reported the standard deviations to show its statistical significance, for both algorithm performances as Tab.1,2,4 and Fig. 3 and time consumption as Tab. 3, also the learning curves in Fig. 4 are shaded by 95% confidence intervals. Other results without standard deviations are adapted from previous papers, where most of the values are reported without statistical significance. The results of BC, BEAR, BRAC, AWR, BCQ, SAC and CQL can be found in D4RL paper and CQL paper, all without standard deviations reported. The results of DT, TD3+BC, Onestep RL, AWAC, IQL can be found in IQL paper, Diffuser and IDQL can be found in IDQL paper, results of $\mathcal{X}$-QL and MoRel can be found in their own papers, these are all reported without standard deviations as well. For Tab. 1 and 2, we manage to find the standard deviations reported for ARQ and already fill the values in the modified draft. The results of SAC, AWAC, ACA in Tab. 4 are also reported without standard deviations in ACA paper. It turns out our paper is the very few papers reporting the statistical significance of our own algorithms in the domain. We appreciate the attention of the reviewer for the statistical significance of the results and we have tried our best to satisfy this requirement.

Performances. We admit the current consistency model does not yield better performances than diffusion models for offline RL settings, and we do not expect it since otherwise the consistency model will not just be an acceleration method of the diffusion model but a total replacement of it. There is always the trade-off of computational efficiency and the model accuracy. We agree that the inference speed is more critical for RL. However, the Tab. 3 may be interpreted in another way by the reviewer as we intended. As also in above general response, in Tab. 3, our major claim is that Consistency-AC can achieve a similar performance as Diffusion-QL with a much smaller $N$, instead of comparing with the same $N$. Consistency-AC saturates its performances with $N=2$ for most of the tasks in our experiments rather than just this one. Specifically, if we look at $N=2$ for Consistency-AC while $N=5$ for Diffusion-QL, the scores are similar but Consistency-AC saves half inference time (1.84/3.76) and training time (31.94/57.06). By default in our experiments, Consistency-AC and Consistency-BC use $N=2$ while Diffusion-BC and Diffusion-QL use $N=5$ for results reported in paper.

Hyperparameter tuning. We may disagree with the reviewer that hyperparameter tuning is making the proposed method less convincing. Actually, the hyperparameters in Tab. 5 are all inherited from Diffusion-QL paper [3] except for the Q norm, some are not reported in the previous paper but can be found in their experiments. Most of the present state-of-the-art algorithms for specific domains require hyperparameter tuning, which is quite common in literature, with evidence in Tab. 3 of paper [3] and Tab. 4 of paper [4]. They may not report full hyperparameters in the algorithms, and some other papers do not even report their hyperparameters, but this does not indicate that there is no hyperparameter tuning in their reported results. We don’t see any reason that our reported results with good hyperparameter tuning indicates the generality of the method is bad. We can understand the concern if the reviewer is asking how robust the proposed method is against each hyperparameter. This can be computationally heavy to evaluate and we do not see a lot of previous work reporting it. This is further answered in later question regarding the choice of gradient norm.

We thank the reviewer for pointing out the mis-reference of Eq. 3, and will modify it in the modified version.

We plan to open-source the code after the review process and the clean-up of the code. We have added the statement in the modified version, thanks for pointing out.

For the questions of different baseline on different domains, those baseline results are adapted from previous Diffusion-QL paper [3] but not cherry-picked by us. We guess one reason that those baselines are filled on Adroit and Kitchen tasks is because some strong baselines (like MoRel, DT, Diffuser, etc) on Gym tasks do not report their values on Adroit and Kitchen tasks while these 'weak' baselines are reported. Our paper just tries to report thoroughly about previous baselines but not take any cherry-picking.

For the question regarding the gradient norm, most of the gradient norm values are directly inherited from Diffusion-QL paper [3] given the usage of the same neural network architectures, which means we do not thoroughly search over the values. Exception exists for Kitchen tasks where we reduce the gradient norm values to stabilize training. However, this is still a rough tuning instead of a thorough searching over the values. A thorough search over hyperparameters can be computationally very expensive. We found the Q value normalization and coefficient $\eta$ to be more influential on performances than gradient norm.

References:

[3] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. "Diffusion policies as an expressive policy class for offline reinforcement learning." arXiv preprint arXiv:2208.06193 (2022).

[4] Garg, Divyansh, et al. "Extreme q-learning: Maxent RL without entropy." arXiv preprint arXiv:2301.02328 (2023).

Thanks a lot for your constructive feedback and insightful questions.

why on many offline and offline2online tasks, the consistency model underperform the diffusion model, while on some online hard tasks, the consistency model seem to be better?

We are very willing to share insights about this phenomenon in future work with more extensive experiments. Our current conjecture is the deficiency within the offline pre-trained models and a straightforward usage of offline models with online data. Future plans involve better ways combining pre-trained models and online data, but it can be beyond the scope of current paper since it mostly focuses on the effectiveness of consistency model from offline data.

statistical significance

We totally agree with the reviewer that the statistical significance is essential for RL. [1][2] are methods not included in current paper but can definitely be added as baselines. However there are lots of methods in this domain and we can only list some most representative ones with limited pages.

gradient norm

The gradient norm is tuned for tasks that are observed to show less stable training progress.

We generally appreciate your comments. However, some additionally required experiments like evaluating the robustness of the models against each hyperparameter, or running some baselines on certain tasks for providing the statistical significance can take lots of additional efforts and computation. If our response has addressed some of your concerns, we would highly appreciate it if you could re-evaluate our work and consider raising the score.

We appreciate the insightful comments and questions from the reviewer.

Less comparable performance for consistency models on halfcheetah-me, kitchen-xxx in Tab. 1 and 2. We admit that there are some tasks showing consistency policies are less performant than diffusion policies, and the computational efficiency is not obtained for free compared with diffusion models at present, as it is an acceleration method without full capability of replacing the diffusion models with a much smaller $N$ yet. However, there are still quite a few tasks showing at least equivalent performances of consistency policies over diffusion policies, like hopper-medium, walker2d-medium, hopper-medium-replay, walker2d-medium-replay, Antmaze tasks on BC setting, and halfcheetah-medium, halfcheetah-medium-replay, walker2d-medium-expert, antmaze-umaze-diverse for actor-critic RL setting.

For the result in Tab. 3, our major claim is that Consistency-AC can achieve a similar performance as Diffusion-QL with a much smaller $N$. Specifically, if we look at $N=2$ for Consistency-AC while $N=5$ for Diffusion-QL, the scores are similar but Consistency-AC saves half inference time (1.84/3.76) and training time (31.94/57.06). This holds for most of the tasks in our experiments, and by default, Consistency-AC and Consistency-BC use $N=2$ while Diffusion-BC and Diffusion-QL use $N=5$ for results reported in paper.

For the first question regarding the multi-modality of the data, the model is not trained across tasks but within each task. Even so, the offline dataset for each task is usually multi-modal since it can be collected by a mixing of behavior policies. For example, the 'medium-replay' dataset in Gym tasks is the replay buffer of a RL policy to reach a medium level performance, which contains samples collected by an evolving policy. In our paper Fig. 5 in Appendix A shows the diversity of the data distribution, although it is not the action distribution given a certain state, the shapes also indicate the complexity of the sample distributions. More details about these tasks are provided in the original D4RL paper. We thank the reviewer for pointing out this problem and will add explanations in the modified version.

For the second question regarding whether large pre-trained representation model will be an alternative of expressive policy, we believe pre-trained visual representation can definitely lead to improved performance for certain domains of RL, e.g., robotic manipulation, but the improvement can source from different aspects compared with the expressiveness of the policies. Pre-trained visual representation is more helpful for providing image information abstraction with semantic understanding. R3M is encoding images as state representation, while consistency/diffusion model represents the conditional distribution of actions given states, so the representation spaces are not the same. If R3M is modified to encode both images and robot actions, while maintaining a multi-modal model, it might be close to the expressive policies. But for current R3M in imitation learning, if the behavior dataset is multi-modal, and the downstream policy of R3M is unimodal (like Gaussian), we may not expect it to capture the multi-modality.

We appreciate the insightful comments and questions from the reviewer.

For the performance gap on Kitchen and Adroit tasks, we show that the computational efficiency is not obtained for free compared with diffusion models. The dataset of the two domains has poor coverage (Adroit) or long-term trajectories (Kitchen), which makes them harder to learn. How to improve the performance of consistency models on these tasks requires further investigation.

We appreciate the idea of using a consistency model as a transition model instead of policies and it is definitely worth trying. It could be more interesting to see the consistency model being used for sequence prediction like Decision Diffuser, which requires more expressiveness and multi-modality. In that case the improvement of computational efficiency can be even more significant. We generally expect the significance of leveraging the consistency model for RL to be larger in more complicated models or tasks, and current experiments are showcased on standard D4RL baseline.