Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps

Thank you very much for your detailed review and valuable feedback. We sincerely appreciate the opportunity to address your comments and clarify our work. Below is our point-by-point response:

---

1. Missing citations

Thank you for bringing these recent works to our attention. We will properly cite these related works in the revised version of our paper.

---

2. Comparison with Additional Baselines

To highlight the strengths of our approach, we conducted additional experiments covering a diverse set of benchmarks:

1. Standard RL approaches including off-policy methods such as SAC and offline RL methods such as AWR and BCQ.

2. Offline-to-online training with diffusion policy optimization approaches including QSM, BDPO, and DAC.

2.1 Comparison with Standard RL Approaches

We appreciate the reviewer's concerns on comparing NCDPO with standard RL approaches such as SAC. To fully show the advantage of NCDPO over standard RL methods, we conduct further experiments on a more challenging benchmark, Franka Kitchen benchmark. DCDPO signifcantly performs the offline and off-policy RL baselines across all tasks.

(Maximum possible score: 4.0) Results for addtional baselines are reported in [1].

2.2 Comparison with Diffusion Policy Optimization Baselines (Offline-to-Online)

We conducted additional experiments on the Robomimic benchmark, which is much more challenging than Gym Mujoco due to reward sparsity and long horizon.

Baselines for diffusion policy optimization in offline-to-online manner: We include QSM, BDPO and DAC as diffusion policy optimization baselines. We first run offline training. Then we adapt these approches to online training setting by collecting environment interactions in a replay buffer and then perform model update on replay buffer.

The table below shows that NCDPO clearly outperforms diffusion policy optimization baselines.

---

3. ODE Solvers for Decoising Process

Thank you for this insightful question. To investigate the alternative of using an ODE solver as a deterministic generation process, we investigated the DDIM, a representative deterministic sampler for the denoising process. In general, NCDPO outperforms DDIM+PPO, especially on the Robomimic can task. We believe this is because using stochastic denoising is better at exploration than deterministic denoising and thus leads to better RL results.

Results comparing DDIM+PPO and NCDPO:

---

4. Missing reference to tables

Thank you for pointing this out. We will add the reference to Tables 1 and 2 at line 212 in the revised paper.

---

We hope these responses help address your concerns. We are grateful for your thoughtful suggestions, and we hope you will consider reevaluating our work based on the additional analysis and clarifications provided.

[1] D4RL: DATASETS FOR DEEP DATA-DRIVEN REINFORCEMENT LEARNING

Dear Reviewer ohxP:

Thank you for taking the time to revisit our submission and adjust your final score. We sincerely appreciate the effort you invested in reading our rebuttal and reevaluating our work.

Thank you very much for your thoughtful review and constructive feedback. We sincerely appreciate the opportunity to respond to your comments and to further clarify our work through additional experiments and analysis that highlight the advantages of our proposed method. Please find our point-by-point response below:

---

1. Writing and Presentation

We are grateful for your comments regarding the writing and presentation. In future revisions, we will carefully revise the paper to improve clarity, strengthen the logical flow of the narrative, and enhance the formatting and visual quality of all figures to meet the standards expected at top-tier venues.

---

2. Experiments on More Challenging Benchmarks

We fully agree that evaluating our method on more complex tasks would better demonstrate the strengths of our diffusion-based policy. To this end, we have conducted additional experiments on Robomimic and Franka Kitchen, both of which pose significant challenges due to sparse rewards, long horizons, and multi-stage subtasks. We compare NCDPO with DPPO and a set of additional offline RL methods.

Robomimic

Robomimic is a highly challenging benchmark. Offline RL methods can not achieve high success rates on Robomimic, especially on transport. Our method consistently outperforms DPPO and the additional baselines, with especially notable gains in the most difficult transport setting.

Baseline results are taken from our paper and [1].

Franka Kitchen

We also extended our evaluation to the Franka Kitchen environment using the D4RL dataset. This setting simulates real-world kitchen tasks with multiple, sequential goals under sparse rewards, presenting highly challenging tasks for RL training. Our method surpasses DPPO and all baselines across all subtasks:

Scores of offline RL baselines are reported in [2].

---

3. Experiments on Visual Tasks (Robomimic)

To address the concern about image-based inputs, we have conducted experiments involving vision inputs the most difficult square and transport tasks. NCDPO outperforms DPPO on visual tasks of Robomimic. More results can be found in Appendix Section B.2.

---

4. Comparison with MLP Policies

We note that the fundamental advantage of NCDPO over MLP policies with standard RL (e.g. PPO) is that, when demonstrations are available, diffusion policies are better at modeling diverse behaviors than MLP policies. Here we provide a direct comparison between diffusion policy + NCDPO and MLP policy + PPO on three environments, Google Research Football, Robomimic and Franka Kitchen.

NCDPO consistently outperforms the MLP+PPO baseline across all tasks and scenarios. This is because, on these three benchmarks, it is hard for a MLP policy to properly model the diverse behaviors in the demonstration dataset.

Google Research Football results:

Robomimic results:

Franka Kitchen results:

---

We will incorporate these clarifications, comparisons, and additional results into the main paper in future revisions.

We hope this response has addressed the reviewer’s concerns and provided a clearer picture of the significance and applicability of our work. We would be sincerely grateful if the reviewer would kindly consider reassessing the evaluation of our submission in light of this additional information.

[1] Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [2] D4RL: DATASETS FOR DEEP DATA-DRIVEN REINFORCEMENT LEARNING

Dear Reviewer,

Thank you for taking the time to review our work. We noticed that you submitted a final justification, but did not leave accompanying comments. We hope that our clarifications addressed your earlier feedback. Could you kindly confirm whether there are any remaining concerns that we could still address? If there is anything that remains unclear or unaddressed, we would greatly welcome further discussion or clarification. We truly appreciate your feedback and are happy to provide any additional information that may help.

Thank you very much for your detailed review and insightful feedback. We sincerely appreciate the opportunity to address your comments and clarify the contributions of our work. Please find our point-by-point response below:

---

1. Comparison with Additional Baselines

Comparison with Additional Baselines in Complex Environments:

Thank you for pointing out additional baselines. We make comparison with additional baselines including Decision Diffuser and offline RL methods. We present results on the Franka Kitchen environment, which is much more challenging than Gym Mujoco. As shown below, our method NCDPO outperforms DPPO and additional baselines. We note that offline RL methods struggle in this challenging environment and performs much worse than diffusion policy based approaches.

These results further highlight the effectiveness of our method in more complex settings. Scores for other baselines are reported in [1].

Offline Methods are Orthogonal to Our Method:

We also kindly note that:

• Offline methods are orthogonal to our focus. In this work, we focus on fine-tuning diffusion policies through online interaction with the environment.
• Our method can also be seamlessly integrated with offline methods by using diffusion policies trained in offline manner as the starting point for online fine-tuning.

---

2. Theoretical Analysis

We agree that formal guarantees would further strengthen our work, and we deeply considered this direction during our research.

However, due to lack of relevant works on backpropagating gradients through denoising steps in diffusion policies, we found that a rigorous theoretical analysis exceeds the scope of this submission. Instead, we provide:

• Intuitive justification: Although both NCDPO and DPPO fine-tune the denoising process with PPO, the way they improve denoising steps is fundamentally different. DPPO independently optimizes individual denoising step. In NCDPO, RL loss is computed only on the denoised action and intermediate denoising steps are improved by backpropagating the gradients through the denoising steps. Our design simplifies the RL objective and jointly optimizes all denoising steps, instead of treating them as individual decision-making steps.
• Empirical validation: Our experiment results on training from randomly initialized policies confirm that NCDPO achieves better sample efficiency and final performance. We hypothesize that this is because our design in NCDPO provides clearer and more informative gradients for denoising steps.

---

3. Fine-Tuning on Already Trained Policies

We kindly note that, in our main experiments in Sec 6.2 and 6.3, NCDPO is applied to fine-tune diffusion policies trained with behavior cloning on demonstration datasets. Our results show that NCDPO can continously improve the performance of already trained diffusion policies with high sample efficiency.

---

4. Comparison with Baselines on Random Datasets

We appreciate the thoughtful question regarding the use of behavior cloning on random datasets. We would like to emphasize that since our main focus is to investigate online fine-tuning for diffusion policies, we select challenging settings where the datasets contain structured and diverse demonstrations. The challenging settings require diffusion policies to model diverse behaviors and thus are suitable to carry out our experiments.

When using random datasets for behavior cloning, we believe this would lead to a weak initial policy and is similar to using a randomly initialized policy, as we discussed in Section 4.

---

We thank the reviewer again for the constructive and thoughtful review. We hope our responses have addressed the concerns and helped clarify the contributions and scope of our work.

[1] D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Thank you very much for your detailed review and valuable feedback. We sincerely appreciate the opportunity to address your comments and provide further clarification. Please find our point-by-point response below:

1. Novelty and Similarity to DPPO: We respectfully believe there may be a misunderstanding regarding the distinction between our method and DPPO. Our approach is fundamentally different in the following ways:

   • DPPO applies policy optimization independently to each denoising step.
   • In contrast, our method optimizes only the final denoising step and backpropagates gradients through the entire diffusion process. This design yields clearer and more informative gradients for intermediate steps, leading to improved sample efficiency.

   To the best of our knowledge, this is the first PPO-based fine-tuning method that introduces backpropagation through diffusion timesteps, which we consider the core innovation of our work.

2. Lack of Real-World Experiments: We appreciate the importance of real-world validation and acknowledge this as a valuable direction for future work. However, our focus is on algorithmic innovation, specifically developing a more effective fine-tuning method for diffusion policies. Sim-to-real transfer, while important, is beyond the scope of this work and orthogonal to our core contributions. We believe our results demonstrate the value of the proposed method in simulated settings and provide a strong foundation for further deployment.

3. Additional Results on Franka Kitchen Environment: To further validate the effectiveness of our approach, we conducted additional experiments on the Franka Kitchen environment. This setting simulates real-world kitchen with multiple sequential goals under sparse reward setting, presenting highly challenging tasks for RL training. We compare our approach with DDPO and offline RL methods. NCDPO consistently outperforms DPPO and additional baselines across all tasks. The results are summarized below:

   Task	NCDPO (Ours)	DPPO	SAC	BC	SAC-off	BEAR	BRAC-p	BRAC-v	AWR	BCQ	AlgaeDICE	COL
   kitchen-complete	3.90	3.58	0.0	1.4	0.6	0.0	0.0	0.0	0.0	0.3	0.0	1.8
   kitchen-partial	2.96	2.68	0.0	1.4	0.0	0.5	0.0	0.0	0.6	0.8	0.0	1.9
   kitchen-mixed	3.95	2.90	0.0	1.9	0.1	1.9	0.0	0.0	0.4	0.3	0.1	2.0

Scores for offline RL baselines are reported in [1].

4. Graphics Quality and Layout: Thank you for pointing out this issue. We will work to significantly enhance the visual quality, clarity, and layout of the figures in the revised version.

5. Title is Uninformative: Thank you for this constructive suggestion. We will revise the title to "Noise-Conditioned Diffusion Policy Optimization with Backpropagation Through Diffusion Timesteps", which we believe more accurately reflects the core ideas and contributions of our work.

---

We hope our responses have addressed the concerns raised and clarified the novelty and impact of our contributions. We sincerely hope you will consider re-evaluating our work in light of these clarifications.

[1] D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Dear Reviewer tKab,

Thank you for your thoughtful review. We appreciate the concerns you raised and have provided detailed responses in our rebuttal.

We would be grateful to know whether our clarifications have resolved those points. If anything remains unclear, or if further details would be helpful, we sincerely welcome further discussion. Your feedback is very valuable.

Dear Reviewer tKab,

Thank you very much for your thoughtful and constructive review. Please find our point-by-point response below:

1. Low Presentation Quality: We appreciate you highlighting this concern. Due to the OpenReview policy, we are unable to update the paper during the rebuttal and discussion phases. However, we assure you that we will make substantial improvements to the organization and presentation of the paper in future revisions.

2. Limited Novelty: We would appreciate it if you could specify the grounds for your assessment of “limited novelty.” In particular, could you indicate which aspects of our contribution you believe lack novelty? Our core contribution lies in introducing the backpropagation of policy loss through diffusion timesteps during PPO optimization—an approach that, to the best of our knowledge, has not been previously explored and constitutes a significant methodological advancement. We would be glad to provide further clarification or supporting details if needed.

3. Hardware Experiments: Thank you for your comment. Based on your comment, we understand that the absence of hardware experiments does not affect the paper’s acceptance.

We are sincerely thankful for the constructive suggestions you have provided. However, we respectfully disagree with the assessment of “limited novelty.” We would be happy to provide further clarification if the reviewer could indicate the specific reasons or aspects on which this assessment is based.