Value Diffusion Reinforcement Learning

We would like to thank you for proposing some valuable comments. Below we respond to your concerns:

Q1: The proof in Line 500 ..........

A1: Thanks for your comments. We make the clarification as follows:

(i) The assumption of proof in Line 500 of manuscript is the learning policy converges, which has been presentation in Line 152 (Theorem 1) of original manuscript (i.e., the equality holds when the policy converges).

(ii) For the question (difficult to converge) you concerned, we experimentally observed that it works well on convergence and performance, as shown in Figure 1 and Table 1 of original manuscript. It is noted that the formula $\frac{P_{\theta}(Y_{1:T}|s,a,Y_0)}{q(Y_{1:T}|s,a,Y_0)}=1$ is the tightest case (i.e., an idea lower bound), which likes the extreme case of lower bound of Variational auto-encoder (VAE). In other work, the tightest case (w.r.t lower bound) does not affect the validity of this theorem, in analogy to the bound theory of VAE.

Q2: There are no concrete data for .....

A2: Thanks for your question. The training and inference time of our method (VDRL) and other baselines on three tasks of MuJoCo, i.e., Ant-v3, Humanoid-v3, and Walker2d-v3, are shown in the following tables. Notably, the official code of the QSM and DACER methods are based on the JAX framework but the implementation of other baselines and our method are based on pytorch. In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch[1]. To ensure a fair comparison, we utilize the implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX). Furthermore, the number of diffusion steps for all diffusion-based online RL baselines (DIPO, QSM, QVPO, and DACER) is set to 20.

Table 1: The training time (h) comparison on three tasks of MuJoCo Benchmarks.

Table 2: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.

From the results of above tables, we can observe:

(i) The training and inference time of our method is faster than the diffusion-based online RL methods (i.e., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-based online RL baselines (T=20) reduce the computational complexity;

(ii) All diffusion-based RL methods (i.e., DIPO, QSM, QVPO, DACER, VDRL) are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e., SAC, TD3, and PPO) ones, due to the iterative denoising process with some steps. Although VDRL is slower than the traditional or distributional RL methods in inference, the inference times (3.7ms) is still tolerable in the real-time application. As discussed in QVPO, most existing real robots only require a 50-Hz control policy, i.e. output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework if it is necessary, like the official code of QSM and DACER methods. Hence, the inference time is not a bottleneck to applying our method to real-time applications.

Overall, the performance improvements of our method are relatively worth with appropriate longer training and inference time.

Q3: The x-axis in Figure 1 should be ...... Can you provide wall-clock training ..........? After changing the x-axis of Figure 1 to wall time, ...

A3: Thanks for your comments. We make the clarification as follows:

(i) The setting of x-axis (training steps) in Figure 1 of our work is the same as the existing diffusion-based online RL methods as well as the traditional online RL methods, including DIPO, QSM, and QVPO, SAC. So the comparison of performance for VDRL and baselines are reasonable and fair.

(ii) For the comparison of training and inference time for VDRL and baselines, please refer to the A2.

(iii) According to your suggestion, here we conduct experiments to compare VDRL and baselines on three tasks of MuJoCo benchmark with the function of wall clock time. Table 3 shows the mean returns and standard deviations for our method (VDRL) and baselines, which further demonstrate that VDRL achieve better or competitive performance compared with these baselines.

Table 3 The final performance of VDRL and baselines on three tasks of MuJoCo benchmark over 8 hours training time.

Q4: Line 40 mentions quantile-based methods, ........?

A4: Thanks for your comments. Following your suggestion, here we conduct experiments on ten Atari games using the Arcade Learning Environment to compare our method with four quantile-based methods, including C51, IQN, ER-DQN, and FQF. The raw scores of these methods are shown in the Table 4, which demonstrate that VDRL achieves better or competitive performance on ten Atari games.

Table 3: The comparison of raw scores between VDRL and four baselines across ten Atari games, starting with 30 no-op actions.

Q5: Moreover, I just noticed a paper... Could you clarify the differences .........

A5: Thanks for your comments. We make the clarification as below:

(i) The paper was not yet publicly available (this paper was accepted in July 1st) when our paper was submitted (our work was submitted in May 15th). Hence, this paper was not cited in our work.

(ii) The differences of our method from the paper DSAC-D are obvious: Our method (VDRL) establishes a theoretical foundation between the variational bound objective (VBO) of diffusion model and value distribution learning objective (VDLO). Building upon this theoretical foundation, we propose our method with optimal process and sample selection, which distinguishes from DSAC-D you said.

Q6: I noticed that the performance ..................

A6: Thanks for your comments. We clarify this question from the following three aspects:

(i) The difference of performance between DACER [10] and our method comes from different implementation settings. As shown in Figure 1 of DACER, the x-axis is not the online interaction steps as in our work (i.e., training step) but the training iteration with 20 interaction steps. Thus, to perform fairly comparison between DACER and VDRL, we need to multiply all the coordinates on the x-axis by 20. With this operation, you will obtain results similar to ours on DACER.

(ii) Our setting strictly follows the existing works: SAC, TD3, DIPO, QSM, QVPO, etc. Our experimental results on all other baselines including SAC are also consistent with these works.

(iii) Following your suggestion, we run experiments on three tasks of MuJoCo benchmark by using the official code of DACER (JAX), as shown in Table 5. In particular, to perform fair comparison, we adjust one training iteration with 1 online interaction step, which is same as the implementation setting of most existing works (i.e., SAC, TD3, DIPO, QVPO).The performance of DACER (JAX) is similar as the experimental results of DACER (Pytorch) in the Table 1 of the original manuscript.

Table 5 The performance of DACER on three tasks of MuJoCo benchmark, which shows that the best mean returns and standard deviations over 800K training steps.

[1] Shutong Ding, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization. NIPS, 2024.

Thanks to the author for the response. Some issues have been resolved. I recently noticed that DIME [1] published at ICML25 is your direct competitor. Please reproduce its results on mujoco or weaken your SOTA claim. I have raised the score to 3.

[1] Celik et al. DIME: Diffusion-Based Maximum Entropy Reinforcement Learning. ICML 2025

We would like to thank you for recognizing our innovation and proposing some valuable comments. Below we make the responses to your concerns:

Q1: Lines 171-174: The statements of underestimation and overestimation are not correct. These estimation biases have different effects in different task settings [1]. Additionally, it is possible to control the estimation bias via maxmin value estimation.

A1: Thanks for your comments. We make the clarification as follows:

(i) For the first sentence of lines 171-174, we conduct additional experiments to verify the effectiveness of the double value diffusion learning rule on three tasks of MuJoCo benchmark, i.e., Ant-v3, Hopper-v3, and Walker2d-v3, as shown in the Table 1 and 2. The result of Table 1 indicates that this learning rule can further reduce value overestimation, and induce a relatively minor value underestimation in some tasks.

(ii) From the experimental results, we can observe that the slight value underestimation induced by this learning rule could improve the performance on the MuJoCo benchmark, which may because the underestimated value do not propagate explicitly through policy update during training. Besides, the effect of underestimation and overestimation, i.e., the second sentence of lines 171-174, have also been similarly discussed in TD3 [1] and DSACT [2].

(iii) As you suggested and introduced in Maxmin Q-learning [3], there are different effects of these estimation bias for distinct tasks. Thus, one promising future direction is to integrate the technique (maxmin value estimation) for different environment settings.

Table 1: The comparison of average value estimation bias between VDRL with and without the double value diffusion learning rule.

Table 2: The comparison of final performance between VDRL with and without the double value diffusion mechanism.

Q2: There are no related experiments of value estimation accuracy that validate the paper's statements: "utilizing the generative capacity of diffusion models to represent expressive and multimodal value distribution". Experimental results on value estimation accuracy.

A2: We make a clarification for the question you concerned. The experimental results of value estimation accuracy were provided in Table 2 of the original manuscript. The table shows the average value estimation bias, i.e., the difference between the estimated Q-value and the true Q-value, for VDRL, traditional model-free online RL, and distributional RL methods. The results show that VDRL consistently achieves lower bias than DASC and DSACT, highlighting the advantages of leveraging diffusion models to capture multimodal and complex value distributions for improved estimation accuracy.

Q3: For coherency of reading, the related work section is usually placed after the introduction section or before the conclusion section.

A3: Thanks for your suggestion. To improve the reading coherence, we will adjust the related work section to precede the conclusion section in the final version.

Q4: Could the authors provide the training and inference speed comparison? One of my concerns is the computational budget of the diffusion value function.

A4: Thanks for your question. The training and inference time of our method (VDRL) and other baselines on three tasks of MuJoCo, i.e., Ant-v3, Humanoid-v3, and Walker2d-v3, are shown in the following tables. Notably, the official code of the QSM and DACER methods are based on the JAX framework but the implementation of other baselines and our method are based on pytorch. In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch [4]. To ensure a fair comparison, we utilize the implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX). Furthermore, the number of diffusion steps for all diffusion-based online RL baselines (DIPO, QSM, QVPO, and DACER) is set to 20.

Table 3: The training time (h) comparison on three tasks of MuJoCo Benchmarks.

Table 4: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.

From the results of above tables, we can observe:

(i) The training and inference time of our method is faster than the diffusion-based online RL methods (i.e., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-based online RL baselines (T=20) reduce the computational complexity;

(ii) All diffusion-based RL methods (i.e., DIPO, QSM, QVPO, DACER, VDRL) are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e., SAC, TD3, and PPO) ones, due to the iterative denoising process with some steps. Although VDRL is slower than the traditional or distributional RL methods in inference, the inference times (3.7ms) is still tolerable in the real-time application. As discussed in QVPO [1], most existing real robots only require a 50-Hz control policy, i.e. output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework if it is necessary, like the official code of QSM and DACER methods. Hence, the inference time is not a bottleneck to applying our method to real-time applications.

Q5: Experimental results on discrete control tasks (e.g., Atari-10). Comparison of the proposed diffusion value method with the IQN.

A5: According to your suggestion, we conduct experiments on 10 Atari games using the Arcade Learning Environment to compare our method with the IQN [5]. The raw scores of VDRL and IQN are shown in the Table 5, which demonstrate that our proposed diffusion value method achieves better or competitive performance on ten Atari games.

Table 5. The comparison of raw scores between VDRL and IQN across ten Atari games, starting with 30 no-op actions, where the reference value from IQN [5].

[1] Scott Fujimoto, et al. Addressing function approximation error in actor-critic methods. ICML, 2018.

[2] Jingliang Duan, et al. stributional soft actor-critic with three refinements. TPAMI, 2025.

[3] Qingfeng Lan, et al. Maxmin q-learning: Controlling the estimation bias of q-learning. ICML, 2020

[4] Shutong Ding, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization. NIPS, 2024.

[5] Will Dabney, et al. Implicit quantile networks for distributional reinforcement learning. ICML, 2018.

We would like to thank you for proposing some valuable comments. Below we respond to your concerns:

Weakness 1

A: Thanks for your comments. We make the clarification as below:

(i) The core theoretical contribution of Theorem 1 lies in establishing the formal connection between the Variational Bound Objective (VBO) of diffusion models and the Value Distribution Learning Objective (VDLO) in distributional RL, inspired by DDPM variational bounds but different from it. i) solved question is different; ii) The whole derivation process is different from the derivation of DDPM variational bounds; iii) The style of variational bounds, such as DDPM, VAE and ours, look similar, which might be the understanding you think "Theorem 1 follows directly from standard DDPM variational bounds".

(ii) We conduct experiments to compare VDRL and baselines on three tasks of MuJoCo benchmark with 10 runs and different random seeds. The Table 1 demonstrates the consistent conclusion claimed in the manuscript.

Table 1: The final performance of VDRL and baselines on three tasks of MuJoCo benchmark with ten seeds over 800,000 training steps.

(iii) For the inconsistency of results between our work and QVPO, we make the clarification that all comparison results of QVPO on MuJoCo benchmark are strictly produced from the official code (https://github.com/wadx2019/qvpo).

Weakness 2

A: Thanks for your review. We make sure to correct these typos and summarize these hyperparameters into Table 3 of manuscript in the final version, according to your suggestion.

Weakness 3 (Part 1) and Question 4

A: Thanks for your comments. The MuJoCo environments have the standard protocols for evaluating various algorithms. Previous online RL methods follows the protocols. For fair comparisons, we also take the same configurations. We very appreciate these points you suggested, which is interesting and promising future direction. But they seems beyond the scope of this work. Due to time limitation, here we conduct an experiment with adding some noises to observation. We observe that noise can disrupt value learning in some tasks by distorting state-action representations. Techniques like data augmentation or adversarial training could improve noise resilience, and we'll explore these ideas in future work.

Weakness 3 (Part 2)

A: Thanks for your question. The training and inference time of our method and other baselines on three tasks of MuJoCo, are shown in the following tables. Notably, the official code of the QSM and DACER methods are based on the JAX framework but the implementation of other baselines and our method are based on pytorch. In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch[1]. To ensure a fair comparison, we utilize the implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX).

Table 2: The training time (h) comparison on three tasks of MuJoCo Benchmarks.

Table 3: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.

From the results of above tables, we can observe:

(i) The training and inference time of our method is faster than the diffusion-based online RL methods (i.e., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-based online RL baselines (T=20) reduce the computational complexity;

(ii) All diffusion-based RL methods (i.e., DIPO, QSM, QVPO, DACER, VDRL) are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e., SAC, TD3, and PPO) ones, due to the iterative denoising process with some steps. Although VDRL is slower than the traditional or distributional RL methods in inference, the inference times (3.7ms) is still tolerable in the real-time application. As discussed in QVPO, most existing real robots only require a 50-Hz control policy, i.e. output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework if it is necessary, like the official code of QSM and DACER methods.

Overall, the performance improvements of our method are relatively worth with appropriate longer training and inference time.

Weakness 4

A: Thanks for your comments. We make the clarification as follows:

(i) Main difference: Our work is to derive and build value-distribution diffusion model, while these works (QSM and QVPO you mentioned) falls in the category of policy-based diffusion models. That is, VDRL is fundamentally different from these diffusion-based policy RL methods you said.

(ii) The core conceptual novelty of our work lies in establishing the (both theoretical and methodological) connection between the Variational Bound Objective (VBO) in diffusion models and the Value Distribution Learning Objective (VDLO) in distributional RL. Crucially, we theoretical prove that the VBO serves as a tight lower bound of the VDLO under KL-divergence minimization. This theoretical foundation underpins our methodological contribution.

Question 1

A: Thanks for your comments. We make the clarification as below:

(i) The MuJoCo locomotion benchmark is widely-used by the existing online RL methods to verify their performance. In practice, we experimentally fit the values on this benchmark, observe that the value distribution is not unimodal.

(ii) We conduct experiments on 10 inherently multimodal Atari games to verify the effectiveness of VDRL compared with four methods, as shown in Table 4. The results demonstrate that VDRL achieves better or competitive performance on ten Atari games.

Table 4: The comparison of raw scores between VDRL and four baselines across ten Atari games, starting with 30 no-op actions.

Question 2

A: Thanks for your comments. We clarify the distinctions as follows:

(i) Theoretical objective and derivation: Our optimization objective is derived from the value distribution learning (i.e., Theorem 1 of Section 3.1 in the original manuscript), while the objective of QSM is the alignment of the score of learning policy. The difference of derivation and formulation for our method and QVPO was introduced in Remark 1 (Line 154-160) of original manuscript.

(ii) Empirical Comparison: The empirical comparison of VDRL, QSM and QVPO on MuJoCo benchmark have been provided in Figure 1 and Table 1 of original manuscript.

Question 3

A: We conduct experiments to compare VDRL, DSAC, and DSACT on three tasks of MuJoCo benchmark with the function of wall clock time.

(i) The Table 5 further demonstrates that VDRL achieve better or competitive performance compared with these methods.

Table 5 The final performance of VDRL and baselines on three tasks of MuJoCo benchmark over 8 hours training time.

(ii) For the comparison of total GPU-hours (training time) for VDRL and these baselines , please refer to the answer of Weakness 3 (Part 2).

[1] Shutong Ding, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization. NIPS, 2024.

We would like to thank you for recognizing our innovation and proposing some valuable comments. Below we make the responses to your concerns:

Q1: The computational overhead of using diffusion in value estimation ..........

A1: Thanks for your question. The training and inference time of our method (VDRL) and other baselines on three tasks of MuJoCo, i.e., Ant-v3, Humanoid-v3, and Walker2d-v3, are shown in the following tables. Notably, the official code of the QSM and DACER methods are based on the JAX framework but the implementation of other baselines and our method are based on pytorch. In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch [1]. To ensure a fair comparison, we utilize the implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX). Furthermore, the number of diffusion steps for all diffusion-based online RL baselines (DIPO, QSM, QVPO, and DACER) is set to 20.

Table 1: The training time (h) comparison on three tasks of MuJoCo Benchmarks.

Table 2: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.

From the results of above tables, we can observe:

(i) The training and inference time of our method is faster than the diffusion-based online RL methods (i.e., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-based online RL baselines (T=20) reduce the computational complexity;

(ii) All diffusion-based RL methods (i.e., DIPO, QSM, QVPO, DACER, VDRL) are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e., SAC, TD3, and PPO) ones, due to the iterative denoising process with some steps. Although VDRL is slower than the traditional or distributional RL methods in inference, the inference times (3.7ms) is still tolerable in the real-time application. As discussed in QVPO [1], most existing real robots only require a 50-Hz control policy, i.e. output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework if it is necessary, like the official code of QSM and DACER methods. Hence, the inference time is not a bottleneck to applying our method to real-time applications.

Q2: While MuJoCo is a representative, standard, and popular benchmark, it remains unclear how VDRL would perform in other tasks.

A2: Thanks for your comments. Following your suggestion, here we conduct additional experiments on 10 inherently multimodal Atari games using the Arcade Learning Environment to verify the effectiveness of VDRL. The raw scores of these methods are shown in the Table 3, which demonstrate that VDRL achieves better or competitive performance on 10 Atari games.

Table 3: The comparison of raw scores between VDRL and four baselines across ten Atari games, starting with 30 no-op actions, where the reference value from C51 [2 ], IQN [3 ], ER-DQN [4], and FQF [5].

Q3: Sensitivity analysis was only conducted on Humanoid-v3 task as an example. However, it remains unclear if the same conclusions holds on the other tasks. The authors should provide ablations on different environments (say three of them) to confirm the observation. Can the authors expand the sensitivity analysis to at least two or three additional environments to confirm the consistency of conclusions?

A3: Thanks for your comment. Following your suggestion, here we conduct additional experiments on three tasks of MuJoCo benchmark, including Ant-v3, Hopper-v3, and Walker2d-v3, as shown in Table 4 and 5. These results further confirm the consistency of conclusions:

(i) the performance does not improve monotonically with an increasing number of diffusion steps, and the diffusion steps should be set to 10 by considering the trade-off between performance and computational complexity;

(ii) the cosine and variance_preserve noise schedules perform similarly, and both outperform the linear schedule. Hence, we choose cosine schedule for all tasks.

Table 4: The final performance of mean return and standard deviations for different diffusion steps.

Table 5: The final performance of mean return and standard deviations for different diffusion noise schedule.

Q4: How critical is the double value diffusion mechanism? Could the authors provide an ablation showing performance with and without this?

A4: Thanks for your comments. To investigate the significance of the double value diffusion mechanism, here we conduct ablation experiments (as you suggested) on three tasks of MuJoCo benchmark, i.e., Ant-v3, Hopper-v3, and Walker2d-v3, as shown in the Table 6 and 7. The results show that VDRL with the double value diffusion mechanism achieves better performance than VDRL without this mechanism in terms of average returns and variance, which verifies the effectiveness of the double value diffusion mechanism.

Table 6: The comparison of final performance between VDRL with and without the double value diffusion mechanism.

Table 7: The comparison of average value estimation bias between VDRL with and without the double value diffusion learning rule.

[1] Shutong Ding, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization. NIPS, 2024.

[2] Marc G Bellemare, et al. A distributional perspective on reinforcement learning. ICML, 2017.

[3] Will Dabney, et al. Implicit quantile networks for distributional reinforcement learning. ICML, 2018.

[4] Mark Rowland, et al. Statistics and samples in distributional reinforcement learning. ICML, 2019.

[5] Derek Yang, et al. Fully parameterized quantile function for distributional reinforcement learning. NIPS, 2019.

Are there any further questions for the authors?

Dear Reviewer yrGJ,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript and providing valuable feedback. We have provided detailed responses to your comments and hope that they adequately address your concerns.

As the discussion stage deadline approaches, we would be grateful to know whether our responses have resolved the issues you raised. If you need further clarification or have any other questions, please feel free to discuss them with us! We are more than willing to continue our communication with you.

Thank you once again for your thoughtful review. We would be deeply appreciative if you could re-evaluate our submission.

Authors