Uncertainty-aware Preference Alignment for Diffusion Policies

Dear Reviewer, we sincerely value your time and effort in evaluating our work. We have prepared comprehensive responses and clarifications to address each point you raised. We hope these responses can resolve your concerns.

Q1. The authors could conduct more ablation studies or expand the range of benchmarks (e.g. medium-replay data in D4RL benchmarks).

A1. Thanks for your valuable suggestions. we conducted additional experiments, including more tasks and ablation studies. Please refer to E.1 and E.3 at https://anonymous.4open.science/r/Diff-UAPA-Rebuttal.

---

Q2. Could you clarify how the two main methodological components (MLE-based preference alignment and the Beta prior) differ in this work compared to [1][2].

A2. Thank you for raising the question. We would like to provide a more detailed discussion highlighting the differences between this paper and the two prior works as follows.

1. Difference between [1]. We would like to highlight that the alignment for diffusion policy via MLE is not our primary contribution. Instead, we primally follow the approach developed for LLM in [1], and adapt it to the RL setting in our work. While we adopt some techniques from [1] (we have correctly cited [1]), the problem setting, preference model, and derivation are different. Specifically,

   • Problem setting. [1] is formulated in the context of LLM alignment, where rewards are assigned exclusively at the final step, and preferences are based solely on the final output of the LLM. This approach optimizes the LLM’s ultimate output (see Eq. 14 in [1]). In contrast, Eq. 12 in this paper extends the framework to a trajectory-wise setting within the RL field. The key distinction is that in RL, we incorporate intermediate rewards. As a result, this paper formulates preference alignment based on the entire trajectory rather than focusing solely on the final state-action pair.

   • Preference model and deviation. [1] employs a reward-based preference model while regularizing the KL-divergence with respect to a reference policy (see Eq. 3 in [1]). In contrast, this work adopts a regret (advantage)-based preference model (see Eq. 5) within the maximum entropy RL framework. To achieve trajectory-wise alignment under the advantage-based preference model, we define the chain advantage function (i.e., Eq. 8) and compute its expected value with respect to the diffusion latent variable.

2. Difference between [2]. While both works utilize the Beta prior with a MAP objective to model uncertainty during the alignment process, this work differs significantly regarding the problem formulation, motivation, and approach to incorporating the Beta prior. Specifically,

   • Problem formulation and motivation. [2] addresses epistemic uncertainty from an offline preference dataset with imbalanced comparison frequencies across trajectories, where fewer compared trajectories induce greater uncertainty in reward prediction. In contrast, our work targets aleatoric uncertainty in human preferences, which arises from the inconsistent preferences from different annotator groups for the same trajectory pair. In other words, by interpreting the parameters $\alpha$ and $\beta$ of the Beta distribution as counts of 'vote' and 'unvote' feedback, [1] models the difference in their absolute values across trajectories (e.g., Beta(10,2) vs. Beta(100,20), where the former shows greater uncertainty due to fewer counts). In contrast, this work uses the Beta prior to model the relative strength of $\alpha$ and $\beta$ for different $\tau$ (e.g., Beta(6,6) vs. Beta(10,2), where the former shows greater uncertainty due to vote inconsistency).

   • Approach. [2] adopts a two-step procedure, proposing a MAP objective for learning a distributional reward model. To achieve this, [2] introduces an iterative update rule that refines the reward model using the learned Beta model, which is then used for policy learning. In contrast, this work derives a unified MAP objective for directly aligning the diffusion policy in a single-step process. By maximizing the likelihood of the diffusion policy's output under the learned Beta distribution, the process guides the policy to align the estimated $\phi(\tau)$ with their prior distribution $p_0(\phi(\tau))$, which is more straightforward and efficient.

---

Q3. It would be more user-friendly if they included a README with clear instructions on how to run the code.

A3. We have included a README file to assist with running the code. Please check https://anonymous.4open.science/r/Diff-UAPA.

---

Q4. Some typos.

A4. Thank you for your valuable suggestions. We have corrected them accordingly.

---

References

[1] Diffusion model alignment using direct preference optimization. CVPR 2024

[2] A Distributional Approach to Uncertainty-Aware Preference Alignment Using Offline Demonstrations. ICLR 2025.

Dear Reviewer, we sincerely value your time and effort in evaluating our work. We have prepared comprehensive responses and clarifications to address each point you raised. We hope these responses can resolve your concerns.

Q1. There are many methods designed to be robust against noisy preferences , including these robust preference learning baselines (e.g., data filtering, label smoothing, and robust loss functions) would improve the experiments. In addition, the paper does not explore different types of noisy preference setups.

A1. We sincerely thank the reviewer for highlighting this valuable concern. In response to your suggestions, we have conducted additional experiments across three distinct noisy preference setups as described in [1], including stochastic noise, myopic noise, and irrational noise. We also incorporated a wider range of baseline methods that are robust to noisy preferences, such as $R^3M$ [1], RIME [2], and UA-PbRL [3]. For detailed results and discussions, please refer to Section E.4 at https://anonymous.4open.science/r/Diff-UAPA-Rebuttal.

---

Q2. There are several papers addressing noisy preference labels.

A2. Thank you for raising this concern. We would like to emphasize that, as defined in Definition 3.1, this paper considers the iterative preference alignment setting, where the preference dataset is updated in each round (potentially with inconsistencies), thus requiring the learned policy to adapt to the new preferences progressively. As shown in Proposition 4.1, the proposed prior model could capture the uncertainty within the process. In contrast, prior works on robust PbRL typically assume a static preference dataset without updates, making them not directly work for this iterative setting.

However, we acknowledge the importance of robustness and its tight relationship with uncertainty. In the revised paper, we have expanded the related works section to provide a more detailed discussion of existing studies on robustness in the context of noisy preference labels, including data filtering, label smoothing, and robust loss functions [1-7].

---

Q3. The proposed method incorporates multiple existing components, such as using a diffusion policy instead of a simple MLP and a beta prior instead of a uniform prior. A more detailed ablation study with a decoupled analysis of each effect would help clarify the impact of these choices.

A3. Thank you for your valuable suggestions. We acknowledge the importance of a more detailed ablation study and analysis, and have conducted ablation studies on the two components individually. For further details, please refer to Section E.3 at https://anonymous.4open.science/r/Diff-UAPA-Rebuttal.

---

Q4. The use of a diffusion policy and beta prior introduces computational overhead. A head-to-head comparison with other baselines would provide a clearer assessment of the method’s effectiveness.

A4. Thank you for raising this concern. The additional computational overhead can primarily be attributed to the following components:

• Diffusion policy. While diffusion policies incur higher computational costs than simpler architectures like MLPs, this overhead is partially offset by the action sequence prediction strategy in [8]. More importantly, diffusion models are widely adopted in RL for their strong generative capabilities and superior performance. In practice, training time for diffusion is roughly twice that of the transformer in our experiments.

• Beta model. In this work, we use efficient techniques like the reparameterization trick to improve scalability. In practice, the computational cost of training the Beta model is similar to training a reward model in traditional PbRL. Since our method avoids training a reward model, the added cost is less effective compared to conventional PbRL. Additionally, the extra computational cost only slightly increases training time—by a few minutes—while the subsequent RL phase is much more demanding, often taking several hours.

---

Q5. Some typos.

A5. Thank you for your valuable suggestions. We have corrected them accordingly and thoroughly checked the paper to avoid similar issues.

---

References

[1] Robust reinforcement learning from corrupted human feedback. NeurIPS 2024.

[2] Rime: Robust preference-based reinforcement learning with noisy preferences. ICML 2024.

[3] A Distributional Approach to Uncertainty-Aware Preference Alignment Using Offline Demonstrations. ICLR 2025.

[4] Corruption robust offline reinforcement learning with human feedback. arXiv:2402.06734.

[5] Sample selection with uncertainty of losses for learning with noisy labels. arXiv:2106.00445.

[6] A note on dpo with noisy preferences & relationship to ipo. 2023,

[7] Distributionally Robust Reinforcement Learning with Human Feedback. arXiv:2503.00539.

[8] Diffusion policy: Visuomotor policy learning via action diffusion. IJRR 2023.

Dear Reviewer, we greatly appreciate your constructive comments. We have seriously considered your suggestions, and we hope the following response can address your concerns:

Q1. The proposed method uses both the demonstration data and the preference data.

A1. Thank you for your comment. As shown in Definition 3.1 and Figure 1, the preference dataset is obtained by comparing samples from the trajectory dataset, so no additional demonstration dataset is required. Our pre-training is performed only with the trajectories in the preference dataset, following standard PbRL practices where pre-training is widely adopted (check "Pretraining" paragraph in [1]). To enhance clarity, we have updated the Introduction section to clarify this point.

---

Q2. One advantage of the proposed method is to "bypass the reward learning"...It is useful to compare the performance of having a reward function.

A2. Thank you for raising this concern. The advantages of bypassing reward learning have been demonstrated in many studies, including robotics [1] and LLMs [2]. To better solve your concern, we conducted additional experiments with two-step PbRL baselines. Please refer to E.2 at https://anonymous.4open.science/r/Diff-UAPA-Rebuttal.

---

Q3. It would be stronger to motivate this work to choose some problem domains that human preferences datasets that are originally inconsistent.

A3. Thank you for highlighting this. The experiments on D4RL (Section 5.2) utilize real human preferences from Uni-RLHF benchmark, which were collected from 100 annotators with diverse backgrounds. We believe these real human datasets are originally inconsistent.

---

Q4. It seems that Eq.12 is a multi-step extension of Eq.2...it would be great to clarify this early.

A4. Sorry for the misunderstanding. $\epsilon_\theta$ represents the optimized diffusion policy parameters, and $\epsilon_{ref}$ represents the reference diffusion parameters. Regarding Eq.12, it can be viewed as a trajectory-wise extension of Eq.2 (step-wise) in the RL setting. To ensure clarity, we have updated the paper with more detailed descriptions when presenting Eq. 2.

---

Q5. Sec.4.2 is hard to follow...It might be helpful to clarify that $\phi(\tau)$ is the random variable, and $p_0(\phi(\tau))(\cdot)$ is the density function.

A5. We apologize for any confusion. As defined in the sentences following Eq. 14 and Eq. 15, $\phi(\tau)\in(0,1)$ is the probability that a trajectory $\tau$ wins against the average candidate in the dataset, and it is a Bernoulli random variable. The prior $p_0(\phi(\tau))$ is indeed the probability density function of $\phi(\tau)$, which follows a Beta distribution, serving as the conjugate prior for the Bernoulli variable $\phi(\tau)$. It reflects our initial belief on the winning probability of different trajectories. Based on it, $p_0(A^{\pi_\theta}(\tau))$ defines our initial belief on the strength of different $\tau$. We have enhanced our presentation accordingly.

---

Q6. Eq.6 is obtained by plugging Eq.4 into Eq.5...However, everyone who contributes to the preference dataset might have different reference policies.

A6. Thank you for raising the point. We would like to emphasize that, when learning from a single preference dataset, the reference policy acts as a third-party baseline, not necessarily aligned with personal preferences. It can be any fixed policy used for applying constraints or regularization during training. For example, in LLMs, the reference policy is usually obtained via Supervised Fine-Tuning, while in RL, it's often derived from BC.

---

Q7. Eq.13 is an assumption...discuss the motivation.

A7. Thanks for your valuable advice. The assumption is based on the fact that preferences align with the negative discounted regret (i.e., $regret(\tau)=\sum_{t}\gamma^t regret(s_t,a_t)=-\sum_{t}\gamma^t A(s_t,a_t)$). Intuitively, we can define $A(\tau)=\sum_{t}\gamma^t A(s_t,a_t)$ to represent the negative regret $-regret(\tau)$. More details can be found in [1].

---

Q8. Line 278...Bayes rule?

A8. Yes.

---

Q9. Sec.4.3 learns the prior $p_0(\phi(\tau)$...how to plug it into Eq.16 to further optimize the policy?

A9. Thank you for your question. The prior model $p_0(\phi(\tau))$ encodes our learned belief about $\phi(\tau)$ for a given trajectory $\tau$. During policy optimization, for each input $\tau$, the diffusion policy computes $\phi(\tau)$ based on its definition, while the prior model outputs $[\alpha_\tau, \beta_\tau]$, which defines the target Beta distribution of $\phi(\tau)$. By maximizing the likelihood of the diffusion model's output under this Beta distribution, the process guides the policy optimization.

---

References

[1] Contrastive preference learning: Learning from human feedback without reinforcement learning. ICLR 2024.

[2] Direct preference optimization: Your language model is secretly a reward model. NeurIPS 2023.