We appreciate you for the careful reading of our paper and detailed discussions.

> Weakness 1

As you mentioned, it is meaningful to compare with other algorithms that have multimodal characteristics. We chose to compare the performance with two online RL with Diffusion model algorithms mentioned in the related work (DIPO and QSM). In Figure 3 of the author global rebuttal PDF, it can be seen that DACER achieved the best performance.

> Weakness 2

We answer this question in two ways. First, some not-toy tasks are inherently multimodal. Zou et al.[1] proposes a vehicle control scenario in which the policies are multimodal, i.e., in the case where traffic regulations are not taken into account and the ego car is directly behind the front vehicle and there is only one weekly vehicle, it is an optimal policy for the ego car to overtake both from the left and from the right. In this case, only a multimodal policy is able to deal with the situation. We add experiments comparing the action (steering wheel angle) distributions of DSAC and DACER at the bifurcation point. We use the diffusion policy to sample 200 different actions and plot a histogram in Figure 5 of the author global rebuttal PDF. The results indicate that DACER learns a bimodal policy, whereas DSAC can only learn a unimodal distribution for overtaking from one side.

Second, we think multimodality enhances exploration. To assess the multi-style capability of DACER, we use the method proposed in [2]. This method analyze the diversity of state trajectories by computing the information entropy from 1,100 independent simulation episodes. The experimental results are plotted in the author global rebuttal Table 2. The results show that in the Humanoid-v3, Ant-v3, Hopper-v3 and Pusher-v2 tasks, DACER achieves the highest entropy value, surpassing DSAC, SAC, DDPG, TD3, PPO, and TRPO.

> Question 1

This question of yours is one of the most creative points of DACER. The entropy of the diffusion policy is difficult to calculate analytically, which means we cannot use entropy regularization methods like in TRPO to adjust the entropy (the gradient cannot be backpropagated to the diffusion policy). Additionally, we tried to adaptively adjust the single-step denoising coefficient based on the estimated entropy during the reverse diffusion process, but the results were not as good as the current methods.

> Limitation

In Appendix B we discuss related Limitation and future work, which we will put in the main text in the official version.

Reference

[1] Zou et al. " Policy Bifurcation in Safe Reinforcement Learning," arXiv, 2403.

[2] Xiao et al. "Multi-Style Distributional Soft Actor-Critic: Learning a Unified Policy for Diverse Control Behaviors," in IEEE TIV, 2024.

Thank you for insightful review. We report below the changes we are currently making to address your comments.

Q1

Our work emphasizes proposing the reverse diffusion process as a novel policy expression, combinable with existing online RL algorithms. Section 2.2 highlights the method's portability. To prove it, global rebuttal Fig 4 includes extra experiments combining the diffusion policy with SAC.

However, we feel that your suggestion is good and we modifiy the relevant statement in Section 2.2 to

"Yang et al.[1] pioneered this approach by proposing action gradient method. This approach achieves policy improvement by updating the action in the replay buffer through the $\nabla_a Q$, followed by mimicry learning of the action post-update using a diffusion model. However, action gradient increased the additional training time. Furthermore, it is difficult to fully learn both the action gradient and imitation learning steps simultaneously, which also resulted in suboptimal performance of this method in MuJoCo tasks. Penka et al.[2] proposes Q-score matching (QSM), a new methodology for off-policy reinforcement learning that leverages the score-based structure of diffusion model policies to align with the $\nabla_a Q$. This approach aims to overcome the limitations of simple behavior cloning in actor-critic settings by integrating the policy's score with the Q-function's action gradient. However, QSM needs to accurately learn $\nabla_a Q$ in most of the action space to achieve optimal guidance. This is difficult to accomplish, resulting in suboptimal performance of QSM."

Q2

As you mentioned, comparing the performance with DIPO and QSM is meaningful. In fact, we have previously conducted relevant experiments. However, using the hyperparameters provided in their papers and testing within our JAX framework, we found that their performance did not surpass SAC as claimed. To avoid any misunderstanding that we intentionally lowered their performance, we chose not to include the experimental results. To address your concerns, we have added the results in the global rebuttal PDF Figure 3. We will proactively contact the authors to discuss why the performance does not exceed SAC.

Q3

First, using GMM as a policy requires pre-specifying the number of Gaussian components, which limits its ability to approximate any continuous distribution. Second, in DACER, entropy estimation updates the noise parameter $\alpha$, where some estimation error is acceptable. Finally, we conduct experiments replacing the diffusion policy with GMM (3 Gaussian components) in MuJoCo tasks, showing that performance is lower than that of the diffusion policy, as illustrated in Figure 4 of the global rebuttal PDF.

Q4

Thank you for your reminder. We have written the revisions for the titles of Figures 3-5 in global rebuttal 4.

Q5

Your concern about training time is understandable, but our starting point is to train in an offline simulation environment and then apply the trained network online. Therefore appropriate longer training time is acceptable if performance gains can be realised.

We add some comparative experiments and the results are shown in Table 1 of the author global rebuttal.

Fortunately, DACER's 1-batch inference time is a small value within 1 ms, which is acceptable in the real-time application. We find DACER's training time is 4-5 times longer than DSAC due to the longer chain gradient of the diffusion policy. Your question is insightful and aligns with our recent work. We plan to introduce an ODE method to eliminate this bottleneck.

Q6

Thank you for your careful discovery that the DAC and DACER legends were written backwards here, which we have corrected.

Q7

As you said, it's important to have an experiment that clearly characterizes the need for a multi-peak policy. We choose the vehicle bifurcation experimental task presented in [3] (like push-T) and plot the action distribution at the bifurcation point in Figure 5 of global rebuttal PDF.

Q8

For the sake of fairness in comparison, we maintained the same experimental setup and plotting standards as used in DSAC. In DSAC, each iteration represents one network update, and 20 samples are sampled per iteration.

Q9

Your proposal is strong. To show that adaptive noise injection can achieve optimal performance without parameter tuning, we conducted experiments in three environments, with training curves in Figure 1 of the global rebuttal PDF. The results indicate that while certain linear decay results can perform comparably to adaptive noise injection, most of the results are not as good as adaptive noise injection.

Q10

Thank you for pointing out that we will modify $\alpha$ in the mathematical description of the denoising process to $\omega$.

Q11&12

It does not prevent policy from learning multimodal behavior. First, the diffusion policy's multimodality comes from the reverse diffusion action generation, with added noise just being a little perturbation. Second, experiments show the distance between peaks is generally greater than 0.5, while the injected noise magnitude is within 0.272, decaying to within 0.15 as training progresses.

For the second problem, we used hyperparameters from the DSAC paper for comparison algorithms. For DACER, the diffusion steps, $\lambda$, and $\alpha$ learning rate were consistently set at 20, 0.1, and 0.03 for nearly all experiments.

Picks

Thanks to your suggestions, we will zoom in on the text in the image, add smoothing to the training curves (all curves in global rebuttle have been added), change the title of Table 1 to lowercase.

Reference

[1] Yang et al. "Policy representation via diffusion probability model for reinforcement learning," arXiv, 2305.

[2] Penka et al. "Learning a diffusion model policy from rewards via q-score matching," arXiv, 2312.

[3] Zou et al. " Policy Bifurcation in Safe Reinforcement Learning," arXiv, 2403.

We thank you for the careful reading of our paper and constructive comments in detail.

> Weakness 1

1. As you pointed out, the entropy of GMM does not have a closed-form solution. Eq.15 provides an upper bound for the entropy of GMM [1], which can be used for approximate estimation. We will update this reference accordingly.

2. In Algorithm 1, we propose calculating 1 EM every 10,000 steps, resulting in a computational overhead that can be ignored in terms of overall efficiency.

3. The reason you did not observe a significant performance difference is that the initial and final values of the linear decay are provided by the results of the entropy adjustment method. We add more experimental results for different linear decay range of $\alpha$ values. In Figure 1 of the author global rebuttal PDF, it can be observed that the best performance after parameter tuning only approaches that of our method, while the computational cost of parameter tuning is significantly higher.

> Weakness 2

As you pointed out, the discussion on training and inference efficiency is very interesting. We acknowledge that DACER has longer training and inference time compared to DSAC. But our starting point is to train in an offline simulation environment and then apply the trained network online. Therefore appropriate longer training time is acceptable if performance gains can be realised (the Humanoid-v3 task with the longest training time in Mujoco took only 7 hours in JAX).

We add experiments (Task: Ant-v3) to compare the inference and training time using the JAX framework on the AMD Ryzen Threadripper 3960X 24-Core Processor, NVIDIA 3090Ti. We use the results of 10 calculations to find the mean and standard deviation. The results are shown in Table 1 of the author global rebuttal 1.

Fortunately, DACER's 1-batch inference time is a small value within 1 ms, which is acceptable in the real-time application. We also find that the training time of DACER is 4-5 times longer than that of DSAC. The bottleneck is the longer chain gradient of the diffusion policy. This question you presented is insightful, and it is actually what we are working on recently. We are going to introduce an ODE method to eliminate the bottleneck. We would advise to follow our future work.

> Weakness 3

Based on the above response, the appropriate longer training time relative to DACER is acceptable. As you pointed out, our performance improvements are particularly significant in the most complex state-space tasks, Humanoid-v3 (376-dimensional state, 17-dimensional action) and Ant-v3 (111-dimensional state, 8-dimensional action), demonstrating DACER's strong potential. In other Mujoco tasks, DSAC has already approached the performance ceiling, while DACER still achieves obvious improvements of 276 and 440 in the complex Walker2d-v3 and Hopper-v3 tasks, respectively, and outperforms or matches DSAC in all other tasks. We believe these performance gains are substantial.

> Weakness 4

For the sake of fairness in comparison, we maintained the same experimental setup and plotting standards as used in DSAC [2, 3]. In DSAC, each iteration represents one network update, and 20 samples are sampled per iteration. For the implementation of TRPO and PPO, we have also aligned with DSAC. In their paper, they collect 2,000 samples per iteration, with a mini-batch size of 10 and repeat_num set to 10. This setup achieves a network update corresponding to 20 samples.

> Weakness 5

You may have misunderstood. The multimodality of the action distribution is not limited to the (0, 0) point; it also exists between adjacent peaks (on two diagonals). This is why, in the results of DSAC and TD3, there are issues with flat and poorly learned regions between adjacent peaks, as described in Section 1.1.

To demonstrate the powerful multimodality of DACER, we add an experiment. We select five points requiring multimodal policies: (0.5, 0.5), (0.5, -0.5), (-0.5, -0.5), (-0.5, 0.5), and (0, 0). For each point, we sampled 100 trajectories. The trajectories are plotted in Figure 2 of the author rebuttal PDF. The results show that compared with DSAC, DACER exhibits strong multimodality. This also explains why only the Q-function of DACER can learn the nearly perfectly symmetrical four peaks.

In addition to using Multi-goal task, we use the method from [4] to analyze the diversity of state trajectories for MuJoCo tasks, with results presented in Table 2 of the author global rebuttal 2. This indicator can measure the algorithm's ability to explore the state space. The results show that in the Humanoid-v3, Ant-v3, Hopper-v3 and Pusher-v2 tasks, DACER achieves the highest entropy value, surpassing DSAC, SAC, DDPG, TD3, PPO, and TRPO.

> Question 1

During training, we found that PPO always remains around its initial value and does not improve reward performance. "Multi-goal" is an environment with very sparse rewards, where huge rewards are only provided at four symmetrical peaks. In this environment, the state initialization is mostly near (0, 0). We think that PPO's conservative policy update mechanism is the fundamental reason it performs poorly in this multimodal environment.

Reference

[1] M. F. Huber et al. "On entropy approximation for Gaussian mixture random vectors," IEEE MFI, 2008.

[2] Duan et al. "Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors," IEEE TNNLS, 2021.

[3] Duan et al. "Dsac-t: Distributional soft actor-critic with three refinements," arXiv, 2310.

[4] Xiao et al. "Multi-Style Distributional Soft Actor-Critic: Learning a Unified Policy for Diverse Control Behaviors," in IEEE TIV, 2024.

I appreciate the very detailed response and new experiments. However, my major concerns about the entropy calculation and the long training time remain:

• Could the authors specify which equation from [1] was used to derive the upper bound referenced in Eq. 15? Additionally, please provide an explanation about when this upper bound will be tight. If possible, could the authors include the derivation of the upper bound in the response?

• While I agree that longer training times are acceptable in simulation, a training period that is 4-5 times longer without significant performance improvement raises concerns about the algorithm's efficiency.

• In my opinion, reporting the number of network updates is not a straightforward measure of the training cost, especially considering the authors' starting point is to 'train in an offline simulation environment and then apply the trained network online.' In this case, wall-clock time would be a better measure, as reviewer 3PmU also pointed out.

• I appreciate the inclusion of new evaluations on the diversity of learned behavior. Could the authors provide the equation used to calculate the Information Entropy? Moreover, the advantages of DACER in Table 2 do not appear significant compared to other methods with single-mode Gaussian policy.

[1] M. F. Huber et al. "On entropy approximation for Gaussian mixture random vectors," IEEE MFI, 2008.

We thank you for the careful reading of our paper and constructive comments in detail.

> Weakness 1

Thank you for your important information. We have added a paragraph to introduce image generation with online RL algorithms.

"Diffusion models are widely applied in the field of image generation [1, 2]. Recently, several studies have employed online RL algorithms to fine-tune these models. Black et al. [3] propose an RL-based method for fine-tuning large-scale diffusion models to match potentially non-differentiable reward functions. This approach views the reverse process of diffusion models as a reinforcement learning task with time steps T, where the output of the diffusion model at each step serves both as an action and the observation for the next step. Fan et al. [4] proposed DPOK, a RLHF method for diffusion models in text-to-image generation. Given a reward model, the goal is to fine-tune the parameters of the diffusion model so that the resulting images from the sampling process achieve high rewards. The reward is a function of the final output of the entire diffusion process and, therefore, not be optimized independently across timesteps. The above work utilizes the reverse diffusion process to optimize the maximum reward, but does not learn the Q-function, as well as using the classical Actor-Critic framework."

> Weakness 2

It can be seen that our core comparison algorithm is DSAC. To be consistent with it, we use the highest value in 10% of the iterations. It is worth noting that each point in the experiment is the average of 10 episodes. We also set 5 different random seeds in each experiment. The results in Table 1 are the average and standard deviation of the results of the 5 random seeds. These settings avoid the influence of luck and noise to a certain extent.

But since you have mentioned it, we also give the mean and standard deviation of the 5 seed results averaged over the last 10% of iterations, which are shown in Table 3 of the author global rebuttal 3.

> Weakness 3

As you pointed out, the discussion on training and inference efficiency is very interesting. We acknowledge that DACER has longer training and inference time compared to DSAC. But our starting point is to train in an offline simulation environment and then apply the trained network online. Therefore appropriate longer training time is acceptable if performance gains can be realised (the Humanoid-v3 task with the longest training time in Mujoco took only 7 hours in JAX).

We add experiments (Task: Ant-v3) to compare the inference and training time using the JAX framework on the AMD Ryzen Threadripper 3960X 24-Core Processor, NVIDIA 3090Ti. We use the results of 10 calculations to find the mean and standard deviation. The results are shown in Table 1 of the author global rebuttal 1.

Fortunately, DACER's 1-batch inference time is a small value within 1 ms, which is acceptable in the real-time application. We also find that the training time of DACER is 4-5 times longer than that of DSAC. The bottleneck is the longer chain gradient of the diffusion policy. This question you presented is insightful, and it is actually what we are working on recently. We are going to introduce an ODE method to eliminate the bottleneck. We would advise to follow our future work.

> Weakness 4

For the sake of fairness in comparison, we maintained the same experimental setup and plotting standards as used in DSAC. In DSAC, each iteration represents one network update, and 20 samples are sampled per iteration. For the implementation of TRPO and PPO, we have also aligned with DSAC. In their paper, they collect 2,000 samples per iteration, with a mini-batch size of 10 and repeat_num set to 10. This setup achieves a network update corresponding to 20 samples. For env steps, you need to multiply the horizontal axis by 20.

> Minor note 1

We modify Eq.1 to $J _ \pi=\mathbb{E} _ {(s _ {i\geq t},a _ {i\geq t})\sim\pi}\Big[\sum _ {i=t}^{\infty}\gamma^{i-t}r(s_i, a_i)\Big]$

Eq.2 to $Q(s,a)=\mathbb{E} _ \pi\Big[\sum _ {i=0}^\infty\gamma^ir(s_i, a_i)|s_0=s,a_0=a\Big]$.

> Minor note 2

$\text{min}\mathcal{L} _ {q}(\phi) = \mathbb{E} _ {({s},{a},{s}^{\prime})\sim\mathcal{B}}\left[\left(r(s,a) + \gamma \mathbb{E} _ {s' \sim p, a' \sim \pi}[Q^{\pi}(s',a')]-Q^{\pi} _ {\phi}(s, a)\right)^{2}\right]$

$\text{min}\mathcal{L} _ {\pi_{}}(\theta) = -\mathbb{E} _ {s \sim d _ {\pi},a \sim \pi }[Q^{\pi _ {\mathrm{old}}}(s,a)]$

> Minor note 3

Thanks to your reminder, we modify the figure notes of Figure 3-5 respectively as

"Ablation training curves for the Entropy Regulator mechanism. DAC stands for not using the Entropy Regulator. DACER's performance on Walker2d-v3 is far better than DAC."

"Ablation experiment curves for the Noise factor modulation mechanism. Adaptive tuning of the noise factor based on the estimated entropy achieved the best performance compared to fixing the noise factor or using the adaptive tuning method with initial, end values followed by a linear decay method."

"Ablation experiment curves for the different diffusion steps. The best performance was achieved with diffusion steps equal to 20, in addition to the instability of the training process when equal to 30."

> Limitation

In Appendix B we discuss related Limitation and future work, which we will put in the main text in the official version.

Reference

[1] Xie et al. "Scalable diffusion models with transformers," ICCV, 2023.

[2] Dhariwal et al. "Diffusion models beat gans on image synthesis," NIPS, 2021.

[3] Black et al. "Training Diffusion Models with Reinforcement Learning," ICLR, 2024.

[4] Fan et al. "DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models," NIPS, 2023.