Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization

Many thanks for the careful review and constructive feedback. Please note our global response with the attached PDF. In the modified revision, we expound on the correlation between the dormant ratio and the performance of the policy network in Section 3.3 as shown in the global response.

These things should be able to work for both low-dimensional and image-based observations. Does this algorithm no longer show any benefit in these settings?

In this work, we propose a proximal policy regularization method to stabilize the training of the consistency policy to improve the sample efficiency of visual RL. CP3ER indeed has the potential to be applied to state-based RL tasks. We supplement comparisons with SOTA methods[1]. The results are shown in Table 1 of the attached PDF. Compared to current SOTA methods, CP3ER demonstrates significant advantages in state-based RL tasks, further proving the generalization capability of CP3ER across different RL tasks.

It would be nice to see perhaps some tasks where a policy that can sample multimodal actions shines above MLP-based policies.

Diffusion/consistency models build the policy distribution without modifying the structure of the policy network. Our policy network utilizes a simple MLP network, which is the same as the networks in Consistency-AC [2] and Diffusion-QL [3].

How big are the datasets? Where does the online dataset come from (ie what RL algorithm, how long was it trained, how often do you update the online dataset, etc)

The pre-collected dataset from D4RL (medium-replay-v2) is used as the offline data, which is from the replay buffer of SAC, comprising 1 million transition tuples. For online training, we employ SAC as the baseline and sample data from the replay buffer to train consistency policy with consistency loss. The replay buffer is updated with each interacting step. For Figure 2, we use the same setting for consistency loss.

Why is there a difference between online and offline dormant ratio for halfcheetah and not walker2d? And how can you draw conclusions from this experiment when you only run two environments that seem to get different results?

The dormant ratio change trends in the two subplots are the same, even though there are slight differences in the specific details. We speculate that the diversity of samples and actions in the halfcheetah dataset is lacking, leading the consistency policy to quickly fit the data distribution and overfit during offline training. In contrast, during the online training, due to the continuous updating of data, the policy network remains active. In the walker2d dataset, the diversity of the samples and action is high, requiring a certain training step to fit, resulting in the same as online learning.

Due to dataset limitations (3 medium-replay datasets are available in D4RL), we conduct 3 tasks when analyzing the impact of offline/online settings on the dormant ratio of the policy. To analyze the effects of training losses and observations on the dormant ratio, we perform 5 different tasks, respectively. Curves not included in the main paper are available in the attached PDF, specifically in Fig. B and C.

What AC algorithm does 'Q-loss' refer to?

Q-loss refers to Equation [2] in the paper, which is the loss used to update the policy.

(Figure 2 and 3) I'm having trouble understanding how to interpret these plots or how you come to your conclusion.

When training the consistency policy using Q-loss, the variance in the dormant ratio of the policy network is quite large. This is because, under some random seed settings, the dormant ratio rapidly increases and remains at a relatively high level throughout. Subsequent training hardly changes the number of active neurons in the network, causing the policy to fall into a dormant where it outputs almost the same actions with different state inputs. With Fig. B and C in the attached PDF, this phenomenon indicates the performance descent. Compared to the state, the network's dormant ratio with the image quickly rises to high values with small variance, indicating severe degradation. With Fig. C in the attached PDF, we can observe that the policy degradation phenomenon of Consistency-AC in visual RL is severe.

It would be nice to see a SOTA on-policy method like PPO

For visual RL, sample efficiency is a common issue. As an on-policy method, PPO does not exhibit notable sample efficiency, which is why it has not been extensively studied in visual RL. Instead, there is more focus on off-policy methods, such as shown in [4-7]. Additionally, in state-based RL, PPO performs poorly compared to other methods(Table 1 in the attached PDF). Furthermore, our motivation is to improve sample efficiency in visual RL, we believe there is no need to compare with on-policy methods like PPO in visual RL tasks.

All of the results show normalized scores, but you never explain how they are normalized, which makes them impossible to compare to other papers.

We used the metrics recommended in [8] to analyze and compare the performance of algorithms. Additionally, we provide the unnormalized IQM curves in the appendix (Figure 12, 14, and 16) for comparison with results from other papers.

[1] Boosting Continuous Control with Consistency Policy, AAMAS 2024.

[2] Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning, ICLR 2024

[3] Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023

[4] DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization, ICLR 2024(Figure 2, 11)

[5] Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning, ICLR 2021

[6] TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning, NeurIPS 2023

[7] Stabilizing Off-Policy Deep Reinforcement Learning from Pixels, ICML 2023

[8] Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021.

Thank you for your detailed review and helpful suggestions. Please note the global response and the attached PDF.

The proposed modification is straightforward compared with existing algorithms.

The key challenge lies in addressing the instability during the training of consistency policy. By analyzing changes in the dormant ratio of the policy network, we observed a policy degradation phenomenon and speculate that this may be due to the instability of the data distribution and the score function represented by the Q-function. Therefore, we introduce entropy regularization. From the perspective of RL, our goal is to imbue the policy with specific attributes by modifying the loss, such as maximizing entropy to boost sample efficiency. From the perspective of matching distributions, entropy regularization provides a prior for the target data distribution, with the Q-function weighting this prior, thus stabilizing the optimization objective of the consistency model.

It would be interesting to see how the proposed algorithm performs in state-based RL settings.

CP3ER can indeed be applied to state-based RL. Following [1], we selected 6 challenging tasks from the DeepMind Control Suite(DMC) environment to evaluate the methods. Table 1 in the attached PDF shows the results of different methods within 500k interacting steps. Compared to commonly used RL methods, CP3ER has significant performance advantages and also outperforms the current SOTA diffusion model-based method CPQL.

For the 8 hard DeepMind Control Suite tasks and also Metaworld tasks, the authors compare CP3ER with TACO/ALIX/DrQ-v2 but not with DrM, which was reported in the original DrM paper to achieve good empirical performance on these tasks. This comparison here is missing.

DrM is indeed an excellent method for visual RL. In fact, we considered DrM in the appendix. Figures 14 and 15 show the comparison of DrM on DMC hard tasks. Figures 16 and 17 show the comparison results on Meta-world tasks. We did not include these results in the main part of the paper because when we reproduced DrM results using the official code[2], we could not achieve the performance claimed on Meta-world tasks. To obtain a fairer comparison, we compared our method and the reproduced results with the curves from [3], as shown in Table 2 of the attached PDF. Even compared to the results in the DrM paper, our method achieves comparable or even better performance. We have also included performance curves for several methods on DMC-hard and Meta-world tasks in the attached PDF(Fig. A, the left part shows curves on DMC-hard tasks and the right part shows curves on Meta-world tasks.).

For the analysis (Figures 2, 3, 9), it would be beneficial to include the episodic return in the plots so that readers can understand the numerical performance of the policy in addition to the policy network’s dormant ratio, which is an intrinsic factor not necessarily correlated with the policy’s episodic return.

Thank you very much for your suggestion. We have added an explanation of the relationship between the dormant ratio and policy performance after Section 3.3 in the revised version as shown in the global response.

However, it is important to note that the dormant ratio is a measure of the capacity and the expressiveness of the network, and it is not a sufficient and necessary condition for assessing policy performance. A policy network with a low dormant ratio may have a low episode return. For example, when trained with supervised learning (such as consistency loss), the network typically exhibits a low dormant ratio, but since its goal is to fit the data distribution rather than to maximize average returns, its policy performance may not necessarily be superior. To adequately assess the policy degradation phenomenon caused by Q-loss, we compared the performance of the consistency policy trained with Q-loss to SAC (the online data collection policy required when training with consistency loss). We also compared the dormant ratios of the policy networks in SAC and redrew Figure 2 (Figure B middle and right, and Figure C in the attached PDF). Consistency policy with Q-loss has a higher dormant ratio compared to those trained with consistency loss and SAC.

We revised Figure 3 and 9 from the original paper with average returns as the other axis. The modifications are shown in Figures D and E in the attached PDF. The results from the figures indicate that generally, the higher the dormant ratio of the policy network, the poorer the performance of the policy. Visual RL tasks exacerbate the degradation phenomenon of consistency policies, resulting in poor policy performance.

The conclusion drawn from lines 202-203, “Therefore, we can infer that visual RL will exacerbate the instability of consistency policy training caused by the Q-loss under the actor-critic framework,” does not seem convincing. Even in ordinary policy

In the third part of Section 4, we analyzed consistency policy, particularly whether Consistency-AC is suitable for visual RL tasks. Figure 3 shows the dormant ratio of the policy network under different observations. Compared to state inputs, the network's dormant ratio rapidly rises to very high values with visual inputs, and the variance is small. This indicates that this phenomenon will stably exist, unaffected by random seeds. Compared to state-based RL tasks, we can see that the policy degradation phenomenon of Consistency-AC in visual RL is quite severe. Consistency-AC adopts a Q-loss under the Actor-Critic framework. Thus, visual RL exacerbates the instability of consistency policies. In the modified revision, we add the explanation in 3.3 discussing the relationship between the dormant ratio and the policy performance.

[1] Boosting Continuous Control with Consistency Policy, AAMAS 2024

[2] https://github.com/XuGW-Kevin/DrM

[3] DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization, ICLR 2024

We hope the reply can address your concerns. We sincerely appreciate your recognition of our contribution and vote to accept our work!

Thank you very much for your recognition of our work.

Why using eq. (8) for sampling weight? Can other functions that shares a similar look demonstrated in Fig 4(b) also work?

In fact, Equation (8) is an empirical formula inspired by the Sigmoid function, and it has been improved to meet the desired properties. We aim to sample the most recently collected data with a high probability during the sampling process while sampling older data with a relatively lower probability. If other formulas can satisfy the required properties (Fig. 4(b)), they may also be used to generate sampling weights.

Thank you for your detailed review and valuable comments. Please note the global response with the attached PDF.

The method is not directly applicable to domains with discrete action spaces.

Indeed, our method has not yet been extended to discrete action spaces because we use consistency models to model the policy, and most diffusion/consistency models are currently applied in continuous state spaces. Therefore, our proposed method is also subject to this limitation. However, to extend it to discrete action spaces, one possible approach could be to binarize the discrete actions and then map them to continuous spaces, as suggested in [1]. This could be an interesting and worthwhile direction for future exploration.

The experiments do not compare with other consistency RL methods.

Our proposed method aims to address the issue of sample efficiency in visual RL. To the best of our knowledge, as of the time of paper submission, CP3ER is the only one to apply diffusion/consistency models to online visual RL policy. This is why we have not compared it with other RL methods based on diffusion models or consistency models. In the realm of state-based RL, there are already some methods based on diffusion models or consistency models, and we have compared our method with them. The results are shown in Table 3 of the attached PDF. Following the settings in [2], we compared CP3ER with Consistency-AC and online Diffusion-QL. As can be seen from the results, although our method is designed to visual RL problems, it still demonstrates significant advantages when extended to state-based RL tasks compared to existing methods based on diffusion or consistency models.

Can you explain in detail how dormant rates affect the expression ability of the consistency policy during training? For example, what does a high dormant rate indicate? What does an increasing dormant rate signify? And why?

The dormant ratio of a neural network represents the proportion of neurons that are inactive. The higher dormant ratio indicates fewer active neurons in the neural network, implying the network's capacity and expressiveness are damaged. The policy network's dormant ratio in RL is related to episode return. A higher dormant ratio hints at lazy action in RL with lower episode returns. Conversely, when the performance is good, the policy network is usually more active, and the dormant ratio is usually lower. This phenomenon has been observed in several studies [3-6].

In Sec. 4, Fig. 3, since Q-loss under the actor-critic framework will destabilize the consistency policy training, why was only the Q-loss used during the training process?

In the third part of Section 4, our main objective is to investigate whether consistency policy is suitable for visual RL tasks. Consistency-AC [2], a typical representative of consistency policy, uses only Q-loss during training. Therefore, we maintained the same settings in this part of the experiment. Fig. 3 illustrates the impact of different observations on the dormant ratio of the consistency policy network. The phenomena in the figure show that consistency policy in visual RL faces a very high dormant ratio, making policy training challenging. From Fig. 3, we can also conclude that Consistency-AC is not suitable for visual RL learning tasks.

How do the computational cost and runtime compare with other baselines?

We evaluated the training and inference time of CP3ER separately, comparing it with the methods mentioned in the paper. The GPU used is 2080Ti and the batch size for training is 256. The results are shown in the following table. Compared to TACO, our proposed CP3ER does not show a significant increase in training time and inference time costs. During training, the additional time cost of CP3ER compared to DrQv2 is mainly attributed to the computation of the regularization loss term.

[1] Structured Denoising Diffusion Models in Discrete State-Spaces, ICLR 2023.

[2] Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning, ICLR 2024

[3] DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization, ICLR 2024(Figure 2, 11)

[4] In deep reinforcement learning, a pruned network is a good network, Arxiv 2024(Figure 11)

[5] The Dormant Neuron Phenomenon in Deep Reinforcement Learning, ICML 2023 (Figure 9, 11, 16)

[6] Pretrained Visual Representations in Reinforcement Learning, Arxiv 2024 (Figure3, 4)

We hope the reply can address your concerns. We sincerely appreciate your recognition of our contribution and vote to accept our work!