We appreciate the reviewer’s thoughtful comments. Here are the responses, where we grouped relevant questions together,

Q1: Explanation about the reverse sampling trick.

I found the reverse sampling trick (14) introduced in line 213 hard to understand and seems lacking any evidence or explanations.

We provide the detailed derivations of the reverse sampling trick in our response to Reviewer pogC.

The proposed RSSM loss and original DDPM loss [1] both aim to train a diffusion model to generate data from the target distribution $p_0$. However, DDPM loss requires sampling from $p_0$, which is not possible in online RL since we only know the energy function of $p_0$.

The core novelty of RSSM is a tractable approach to train diffusion model/policy purely from the energy function rather than samples. The reverse sampling trick is an algebric operation to use sampling-based approximation to calculate the loss.

The sampling distribution $\tilde{p}_t$ in Theorem 3.2 is introduced without any explanation, which is very confusion. It is better to, at least explain that, it can be approximated with diffusion samples.

The sampling distribution $\tilde{p}_t$ is a distribution we can choose. Any distributions covering the support $p_t(x_t)$ can be used, such as Gaussian, Cauchy, etc.

Being approximated by diffusion models is not required for $\tilde{p}_t$.

The choice of $\tilde{p}_t$. Our choice of $\tilde{p}_t$ is based on the fact that the RSSM loss relies on the energy function ($Q$-function in RL) that is more accurate as samples of $a$ are close to the current policy. Therefore, we directly use the current reverse sampling at the $t$ step, which is exactly the current policy.

[1] Ho, J., Jain, A., & Abbeel, P. Denoising diffusion probabilistic models. NeurIPS 2020.

Q2:

In section 5.2, the authors do not provide results for DPPO, TD3 or PPO. I think it is better to provide results for all the baselines...

We report the results for all the baselines, the reviewer might mean we did not provide the training curves in Figure 2 for DPPO, TD3, and PPO. We reported in Figure 8 here.

Q3 Performance comparison between TD3 and SAC

because TD3 is reported to be consistently outperforming SAC on Mujoco Tasks.

Could you please analyze why TD3 cannot outperform SAC?

In Table 1,

• TD3 is better in Reacher, Humanoid, Pusher, Ant, and Swimmer.
• SAC is better in HalfCheetah, Hopper, and Walker2d,
• They are very close in InvertedDoublePendulum and InvertedPendulum.

Therefore, it is difficult to say one algorithm outperforms another one.

Generally speaking, the performance gap between TD3 and SAC is sensitive to various hyperparameters. We reviewed some of our references using both as baselines and summarized the comparison in the figure below.

Therefore, it is generally not straightforward to conclude that SAC consistently outperforms TD3, or vice versa.

Q4: about the toy example

The author argue that RSSM can apply to any probablistic model with known enery function. The Gaussian mixture task is way to simple for support this... it seems that the proposed methods are quite restricted to RL problems...

A4: Our toy example on Gaussian mixture is just a proof-of-concept, not for showing the capability limit of RSSM. Moreover, modeling a Gaussian mixture is non-trivial since [1] highlights its slow-mixing issue, and our Figure 2(e) further shows that naive Langevin dynamics fails to recover the correct mixture within finite steps.

To further demonstrate RSSM’s effectiveness, we include results on the Two Moon distribution, a standard benchmark for Boltzmann samplers, in Figure 6. We also add iDEM [2] and FAB [3] as baselines. RSSM performs well and achieves the lowest KL divergence among all methods.

[1] Song, Yang, and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.

[2] Akhound-Sadegh, Tara, et al. Iterated denoising energy matching for sampling from boltzmann densities.

[3] Midgley, Laurence Illing, et al. Flow Annealed Importance Sampling Bootstrap.

---

We appreciate your valuable comments and hope our responses have clarified the concerns raised. We would be grateful if you would consider updating your score in light of these clarifications.

We thank the reviewer for the constructive feedback. Below, we provide our response and have merged some questions and reindexed the references for clarity.

Claims And Evidence

Q1:

reasons why other baseline methods have higher memory/time than SDAC.

benefit compared to other Boltzmann sampling methods such as iDEM [3] and FAB [4]?

1. DQS/iDEM [1,2] use $K$ Monte-Carlo samples and $K$ energy function evaluations for one $\nabla\log(x_t)$. RSSM only needs one sample and energy function evaluation, saving memory and time.
2. The sampling in RSSM is unbiased, while iDEM sampling is biased according to Proposition 1 in [2].
3. DQS/iDEM/QSM use the gradient of energy function, which might be inaccurate when the energy function (Q-function) is learned in RL.
4. FAB [4] leverages flow models whose expressiveness is limited compared to diffusion-based models. FAB also suffers from high variance due to importance sampling.

Add Boltzmann samplers as toy example baseline.

We add iDEM and FAB to the toy example, as well as the two-moon distribution in Figure 5, 6. We also count the training time and memory footprint in Table 5.

The claim [3] “induce huge memory and computation costs” is not elaborated upon sufficiently. [3] updates the actions stored in the replay buffer (NOT as a set of particles as claimed by the authors)...

We respectfully disagree with the reviewer’s claim. According to the official DIPO implementation [3], line 116 in main.py explicitly calls diffusion_memory = DiffusionMemory(state_size, action_size, memory_size, device), clearly indicating the use of an additional buffer, inducing memory and time cost.

• QVPO,DACER,DIPO: discussed in Section 4.2.

---

Q2: Performance not comparable to spinning up SAC at 1 million updates.

A2: Spinning Up’s results run for 3 million updates, not 1M as claimed. At 1M, it often underperforms SDAC, which uses only 200K updates.

---

Methods And Evaluation Criteria

Q3: Experiments on other tasks.

• Please refer to Figure 4 & Table 4. We select three tasks from DeepMind Control Suite, and a PushT task that is common in diffusion policy.

---

Theoritical claims

Q4: RSSM does not satisfy Bayes' rule.

A4: We clarify that the reverse sampling distribution $\tilde q_{0|t}$ is not, and does not intend to approximate the posterior distribution $q_{0|t}$ in the reverse process mentioned by the reviewer. $q_{0|t}$ is usually intractable, while $\tilde q_{0|t}(\cdot|a_t)$ is a Gaussian.

We respectfully argue that we did not ignore any terms. In fact, our derivation does not use the Bayes' rule, it is based on the probability density function of Gaussians. We provide the detailed derivations in our response to Reviewer pogC, including the score function equivalence.

The novelty of RSSM is integrating score matching loss (12) over measure $g$ in (31), so that it is tractable through unbiased sampling-based approximation, when we cannot sample from $p_0$. We realize that the current notations easily cause causality confusion. We will carefully revise the notations in the revision.

---

Experimental Designs

Please see Figure 1 and Table 1 for performance of 1M updates.

Essential References Not Discussed

Q5:

The description of [1] is completely incorrect, in that it does not use Langevin sampling, rather it uses a diffusion process...

the authors fail to acknowledge that several existing approaches [1,2] that tackle these issues.

We acknowledge a partial inaccuracy in our claim about [1]. Nonetheless, one of the core issues we highlight remains: the gradient of learned Q-functions might be inaccurate, affecting RL performance.

We did miss the literature on Boltzmann samplers. We will address this in the revision.

---

Other questions:

Q6 Computing the log probability, the necessity of additive Gaussian noise, and using temperature parameters.

A6 Please refer to A3,A4 in rebuttal to reviewer pogC. We use temperature parameters to control the additive noise scale.

Thanks for pointing out the typos and writing comments. We will revise accordingly.

[1] Jain, Vineet, et al. Sampling from Energy-based Policies using Diffusion.

[2] Akhound-Sadegh, Tara, et al. Iterated denoising energy matching for sampling from boltzmann densities.

[3] Midgley, Laurence Illing, et al. Flow Annealed Importance Sampling Bootstrap.

[4] Yang, Long, et, al. Policy representation via diffusion probability model for reinforcement learning.

---

We appreciate your valuable comments and hope our responses have clarified the concerns raised. We would be grateful if you would consider updating your score in light of these clarifications.

We thank the reviewer for the thoughtful and constructive feedback. Below, we first provide detailed derivations of the reverse sampling trick, followed by detailed answers to the specific questions.

Reverse Sampling Trick Derivations

Check rebuttal_proof if the equations do not show up.

We first explain (14) from Gaussian probability density function (PDF),

$\tilde{q}\_{0|t}(\\tilde a\_0|a\_t)$ is defined as $\mathcal{N}(\tilde a_0;\frac{1}{\sqrt{\bar\alpha_t}}a_t, \frac{1-\bar\alpha_t}{\bar\alpha_t}I)$ with PDF

$$\tilde{q}_{0|t}(\tilde a_0|a_t) = (2\pi\frac{1-\bar\alpha_t}{\bar\alpha_t})^{-d/2}\exp(-\frac{||\tilde a_0 - \frac{1}{\sqrt{\bar\alpha_t}}a_t ||^2}{2\frac{(1-\bar\alpha_t)}{\bar\alpha_t}})$$

The PDF of forward process perturbation kernel $q_{t|0}(a_t|a_0)$ is

$$q_{t|0}(a_t|a_0)=\mathcal{N}(a_t;\sqrt{\bar\alpha_t}{a_0}, (1-\bar\alpha_t)I) = (2\pi(1-\bar\alpha_t))^{-d/2} \exp(-\frac{||\sqrt{\bar\alpha}_ta_0 - a_t ||^2}{2(1-\bar\alpha_t)})$$

Note that $\tilde q_{0|t}$ is NOT the posterior distribution of $q_{t|0}$, which is usually intractable. Notice that

$$\\nabla_{a_t}\log \\tilde{q}\_{0|t}(a_0|a_t) = \\nabla_{a_t}\\log q_{t|0}(a_t|a_0) = - \\frac{a_t - \\sqrt{\bar\\alpha_t}a_0}{1 - \\bar\\alpha_t} \\quad (*)$$

With these properties, we start from (13)

$$\iint \tilde{p}\_t(a_t | s) q\_{t | 0}(a_t | a_0) \exp (Q(s, a_0) / \lambda) ||s_\theta(a_t;s,t) - \nabla_{a_t}\log q_{t|0}(a_t|a_0)||^2 d a_0 d a_t\quad (13)$$

We first substitute in the PDF of $q_{t|0}$,

$$= \iint \tilde{p}\_t(a_t \mid s) (2\pi(1-\bar\alpha_t))^{-d/2} \exp(-\frac{||a_t - \sqrt{\bar\alpha_t}a_0||^2}{2(1-\bar\alpha_t)})\exp (Q(s, a_0) / \lambda) ||s_\theta(a_t;s,t) - \nabla_{a_t}\log q_{t|0}(a_t|a_0)||^2 d a_0 d a_t$$ $$= (\bar\alpha_t)^{d/2}\iint \tilde{p}\_t(a_t \mid s) \underbrace{ (2\pi\frac{1-\bar\alpha_t}{\bar\alpha_t})^{-d/2}\exp(-\frac{||a_0 - \frac{1}{\sqrt{\bar\alpha_t}}a_t ||^2}{2\frac{(1-\bar\alpha_t)}{\bar\alpha_t}}) }_{ \tilde{q}\_{0|t}(a_0|a_t)}\exp (Q(s, a_0) / \lambda) ||s\_\theta(a_t;s,t) - \nabla\_{a_t}\log q\_{t|0}(a_t|a_0)||^2 d a_0 d a_t$$ $$= (\bar\alpha_t)^{d/2}\iint \tilde{p}\_t(a_t | s) \tilde{q}\_{0|t}(a_0|a_t) \exp (Q(s, a_0) / \lambda) ||s_\theta(a_t;s,t) - \nabla_{a_t}\log \tilde{q}_{0|t}(a_0|a_t)||^2 d a_0 d a_t \quad (**)$$

where the final equality we leverage $(*)$.

The $(**)$ equals the equation (15) times constant $\bar\alpha_t^{-d/2}$, not affecting the optimal solution at given $t$.

[1] revealed that ignoring the $t$-scale constant weights does not affect the performance of diffusion models when learned jointly like (16), thus we ignore the constant and sample uniformly from timestep $t$.

[1] Ho, J., Jain, A., & Abbeel, P. Denoising diffusion probabilistic models.

---

Experimental results or analysis

Please refer to Figure 6 for ablation study of $\lambda$ learning rate.

---

Essential References Not Discussed

Q2: Relation to QIPO.

A2: Our loss function looks similar to QIPO. However, the partition function estimation in QIPO is expensive and biased. The proposed SDAC does not need partition function estimation, and our estimation is unbiased.

Moreover, the QIPO paper was published after ICML submission, we will add QIPO to the revision.

---

Other Strengths And Weaknesses

Q3: "efficient behavior policy" trick:

A3: Yes, we use it. We have the same observation as QVPO, the diffusion policy is too random, thus we use the efficient behavior policy trick, sampling multiple actions and selecting the one with the highest Q-function. It is a standard trick in diffusion policy and diffusion-based planners. We skip it due to space limit and will clarify it in the revision.

---

Q4:

Log probability is intractable in diffusion policy, ..., standard PD update without incorporating log probability terms.

A4: As Q3 mentioned, we use the efficient behavior policy trick and additive Gaussian noise. We notice the policy stochasticity is low after the efficient behavior policy trick, thus we can approximate the entropy using only the log density of the additive Gaussian. We provide results fixing the log density term in Table 3.

---

Q5: Entropy update.

A5: Our $\lambda$ updates are adaptive in line 185 of /relax/algorithm/sdac.py. We will fix the typo.

Q6: Performance with 1M updates

A6: We provide 4 envs that are not saturated with 200K updates in Figure 1. SDAC outperforms all the baselines.

---

We appreciate your thoughtful review and hope our responses have clarified the concerns raised. We would be grateful if you would consider updating your score in light of these clarifications.

We thank the reviewer for the thoughtful comments. Here are the responses to the questions,

Theoretical claims

Q1:

How should $\tilde{p_t}$ be chosen? Specifically, what is the appropriate distribution over $t$, and what form should $\tilde{p_t}$ take? it seems that any choice might work for the algorithm—is that correct?

A1: Yes. Theoretically, as long as the support of $\tilde{p}_t$ covers the support of $p_t(a_t)$, i.e., the distribution of the forward process at time $t$, we can recover the noisy score function by optimizing the RSSM loss (9).

We discussed our empirical choice of $\tilde{p}_t$ in Section 4.1. In online RL, RSSM loss depends on the Q-function, which might not be accurate for any action $a$. Therefore, we design $\tilde{p}_t$ by sampling from the reverse process induced by the current policy, ensuring alignment with regions where the Q-function is more reliable.

---

Q2:

Is the assumption in Eq. (10) too strong? Does it limit the model’s expressiveness? Furthermore, does this assumption still yield a valid ELBO? ...

A2: We would like to clarify that the reverse sampling distribution $\tilde q_{0|t}(a_0|a_t)$ in eq.(10) is not the unknown posterior distribution $q_{0|t}(a_0|a_t)$ in the reverse process, or its approximations. We did not change the reverse process, so its expressiveness and the ELBO to learn it are not affected. $\tilde{q}_{0|t}(a_0|a_t)$ is a known distribution $\mathcal{N}(\frac{1}{\sqrt{\bar{\alpha_t}}}a_t, \frac{1-\bar\alpha_t}{\alpha_t}I)$, only used in computing RSSM loss function.

The proposed RSSM loss and original DDPM loss to train diffusion models [1] are both derived from the same ELBO. However, DDPM loss is not tractable when we cannot sample from dataset $p_0$ and only know its energy function, which is the case in online RL. The novelty of RSSM is a tractable sampling-based method to train a diffusion model/policy when only the energy function is available.

---

Q3:

How is entropy estimated using this method?

A3: When we implement the diffusion policy, we leverage a trick called efficient behavior policy, where we sample multiple actions and choose the one with the highest $Q$ function, which is also used in many diffusion policy papers like [1,2]. We notice that after the efficient behavior policy trick, the policy stochasticity is low, so we can estimate the entropy by only the entropy of additive Gaussian noise.

Our strong empirical performance shows that this simple entropy estimation works well enough.

[1] Ding, Shutong, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization.

[2] Janner, Michael, et al. Planning with diffusion for flexible behavior synthesis.

---

Experimental Designs

Q4:

Clarification of reverse sampling distribution $\tilde{q}(a_0 | a_t)$ since action generation still requires reverse sampling, is it possible to directly use $\tilde{q}(a_0 | a_t)$ instead?

A4: As the A2 pointed out, $\tilde{q}_{0|t}$ is not the reverse process posterior, and we did not change the reverse denoising process. Therefore, we still need to sample from the reverse process, which is the key to the rich expressiveness of diffusion models.

---

Q5:

When normalizing Q-values, do you sample multiple actions for a given state, or is the normalization performed within a batch?

A5: We just normalize within the batch, no additional sampling is needed.

---

Q6:

I noticed that DACER exhibits extremely high variance in your experiments. However, this phenomenon was not present in the original DACER paper, which used the v3 dataset. Do you have any insights into why this discrepancy occurs? Could you also provide some experimental results on the v3 dataset for a more straightforward comparison?

A6: Thank you for the keen observations. The differences stem from hyperparameter choices: the original DACER uses a smaller learning rate (1e-4) and 20 parallel environments, resulting in significantly more fresh data and lower variance. In contrast, we use 3e-4 with 5 environments across all algorithms for a fair comparison.

We provide DACER (using both our and original hyperparameters) and SDAC in both V3 and V4 in Figure 2 and Table 2. The original DACER parameters show great stability but much slower learning. The variances of V3 and V4 envs are similar.

The proposed SDAC can achieve stable learning with a larger learning rate and fewer envs (as 1 is commonly used), demonstrating better robustness compared to DACER.

---

We appreciate your thoughtful review and hope our responses have clarified the concerns raised. We would be grateful if you would consider updating your score in light of these clarifications.