Uncertainty-aware Preference Alignment for Diffusion Policies

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. We have carefully addressed each of your comments with detailed responses and clarifications. We hope these will effectively resolve your concerns.

Q1. Although the Beta distribution can theoretically represent uncertainty in the strength of preferences, its effectiveness under complex preference distributions (such as asymmetric or multi-modal cases) remains questionable.

Response. We thank the reviewer for raising this important point. To clarify, we do not use the Beta distribution to model preference strength directly. Instead, as detailed in Lines 213–217, we use a Beta distribution to serve as a prior on the win‑probability $\phi(\tau)=sigmoid(A^\pi(\tau)-\bar{A}^\pi(\tau))$, i.e., the probability that trajectory $\tau$ wins against the average candidate. Since human pairwise feedback is a binary comparison, which is inherently Bernoulli (win vs. loss). So the Beta distribution serves as a natural, conjugate choice: its two parameters flexibly capture both skewed (peaked near 0 or 1) and broad (centered near 0.5) probability without too much complexity. Empirically, even under asymmetric or multi-modal cases, the aggregated win-probabilities remain effectively unimodal and are well-fit by a single Beta. For example, they either concentrated near 0 or 1 when consensus is strong. We acknowledge that truly multi‑modal preference patterns may necessitate more expressive priors such as mixtures of Beta distributions, which is a promising direction for future work.

---

Q2. Additionally, the paper initializes the Beta parameters $\alpha$ and $\beta$ to 1. How would varying the initialization affect the model’s performance under different scenarios (e.g., the various noise conditions shown in Table 5)?

Response. Thank you for raising this question. In this work, we utilize $\alpha_0 = \beta_0 = 1$ as an uninformed prior. This corresponds to a uniform distribution over $[0, 1]$, which serves as a natural choice for uninformed belief. This choice provides a neutral starting point with the least information.

In terms of varying $\alpha_0$ and $\beta_0$ to other values, this operation indeed has an impact on the model performance. For instance, setting $\alpha_0 = 9999$ and $\beta_0 = 1$ will introduce strong prior information into the model (implying 9999 persons vote 1, and only one person votes 0). Moreover, the absolute values of $\alpha_0$ and $\beta_0$ critically affect prior strength: although both $\text{Beta}(1,1)$ and $\text{Beta}(10000,10000)$ have a mean of 0.5 (assuming every trajectory is an average player), the latter encodes a far stronger initial belief in such a “half-win” probability. Such a concentrated prior can dominate the posterior and hinder adaptation to true preference signals. Therefore, we choose the uniform distribution to initialize $\alpha_0 = \beta_0 = 1$ due to the above reasons.

---

Q3. Equation 8 and Equation 17 introduce additional computational overhead. How does this overhead compare to previous methods? It would be helpful if the authors could provide a more detailed comparison of the computational cost relative to prior approaches.

Response. Thank you for your thoughtful comment. We address the computational overhead introduced by the diffusion policy (Eq. 8) and the Beta model (Eq. 17) in Appendix D.3. To summarize, training the diffusion policy takes approximately twice as long as our Transformer baseline. However, this overhead is partially mitigated by the action-sequence prediction strategy adopted from [1], which improves sample efficiency. As for the Beta prior, it is learned using efficient reparameterization techniques, incurring a cost comparable to fitting a standard reward model. In practice, this adds only a few additional minutes to the overall training time.

[1] Diffusion policy: Visuomotor policy learning via action diffusion. International Journal of Robotics Research, 2023.

---

Q4. The comparative experiments only benchmark against diffusion-based methods. Why not compare with non-diffusion baselines (such as INPO, IPO, or Nash-MD-PG)?

Response. Thank you for the comment. We would like to clarify that our evaluation does include non-diffusion baselines. Specifically, as shown in Lines 265–270 and Table 1, there is a line of baselines based on the Behavior Transformer (BET), which is a representative non-diffusion policy optimization method. We assume that the "non-diffusion" approaches you mention (INPO, IPO, Nash-MD-PG) correspond to "non-DPO" methods. Indeed, our experiments also include non-DPO methods like UA-PbRL and RIME (see Lines 336–340 and Table 4).

To solve your concerns, we conduct additional experiments using the baselines you mentioned.

The results highlight the uncertainty-aware capability of Diff-UAPA. In contrast, InPO and IPO perform worse due to the absence of mechanisms for handling uncertainty in noisy labels.

---

Q5. In Equation 11, multiple layers of approximation are introduced. Could the accumulation of these approximations affect the final performance?

Response. Thank you for your valuable question. We assume that the 'approximation' you mentioned indicates the 'inequality' and 'expectation' in Eq. 11. We have discussed some reasons in Lines 193–197, and we have offered a more detailed justification of the necessity and rationale behind our model in Lines 193–197.

To clarify, approximating expectation via sampling is a common practice. When it comes to Jensen's inequality:

• Jensen’s inequality gives an upper bound on the negative log-sigmoid, where the error does not grow infinitely with variance. So training the loss function of Eq. 11 yields similar results to Eq. 10.

• Evaluating Eq. 10 directly means computing probabilities at every diffusion step, which is an expensive process where small errors can be amplified by the outer sigmoid. By applying Jensen’s inequality to the log-expectation, we obtain a simpler surrogate loss that is both faster to compute and more stable.

It is also worth noting that the derivation from Eq. 10 to Eq. 11 follows the standard practice for MLE-based diffusion policy alignment. Similarly, [2] applies an analogous approximation when moving from Eq. 13 to Eq. 14.

[2] Diffusion model alignment using direct preference optimization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

---

Q6. Proposition 4.1 assumes that trajectories are i.i.d, but in practice, trajectories may be dependent. How can you ensure that this is maintained in MDP?

Response. We appreciate the reviewer’s question regarding Proposition 4.1. In fact, Proposition 4.1 rests on the classic Beta–Bernoulli conjugacy, which formally assumes that the $k$ wins out of $n$ comparisons behave like conditionally independent Bernoulli trials given a latent win-probability $\phi$. Crucially, one only needs the weaker condition of exchangeability, and De Finetti’s theorem guarantees that any exchangeable Bernoulli sequence can be viewed as i.i.d. draws from some $\phi \sim \text{Beta}(\alpha_0, \beta_0)$, yielding the posterior $\phi \mid k, n \sim \text{Beta}(\alpha_0 + k, \beta_0 + n - k)$. In episodic RL, independent environment resets, a stationary policy during data collection, and randomized initial states ensure that the sequence of win/loss labels across trajectories is effectively exchangeable (and hence conditionally independent given $\phi$), even though state–action pairs within each trajectory remain correlated by the MDP dynamics. Therefore, Proposition 4.1 does not require trajectories to be strictly i.i.d., and it only requires the exchangeability of the binary preference outcomes, a condition naturally met by standard episodic rollout protocols.

---

Q7. In Algorithm 1, how is the reference policy $、pi_{\text{ref}}$ initialized? How sensitive is the final performance to the choice of $\pi_{\text{ref}}$?

Response. Thanks for your valuable question. $\pi_{\text{ref}}$ is first randomly initialized and then trained via standard behavior cloning on the entire set of state-action pairs extracted from the preference dataset $D_{pref}$, until it reaches a predefined performance threshold (e.g., a target cumulative reward). To solve your concerns, in terms of the sensitivity of the performance $\pi_{\text{ref}}$, we conduct additional experiments by varying the performance of $\pi_{\text{ref}}$ in the Robomimic Lift environment.

From the experimental results, it can be seen that the quality of the initial policy has an impact on the final result. But the Diff-UAPA still outperforms other baseline methods.

---

Q8. Some typos.

Response. Thank you for your valuable suggestions. We have corrected them accordingly and thoroughly checked the paper to avoid similar issues.

Dear Reviewer, we sincerely value your time and effort in evaluating our work. We have prepared comprehensive responses and clarifications to address each point you raised. We hope these responses can resolve your concerns.

---

Q1. This method builds upon several existing works ....

Response. Thank you for raising this concern. We would like to provide a more detailed discussion highlighting the differences between this paper and the three prior works as follows.

1. Difference with [1]. Though both works start from the Maximum Entropy reinforcement learning, [1] uses the cumulative reward of the trajectory as the compare metrics, while ours adopt the advantage function.

   • Preference model. In [1], the preference model employs the cumulative reward of a trajectory as the comparison metric, utilizing an implicit reward model defined as $r(s, a) = \beta \log \frac{\pi(s \mid a)}{\pi_{\text{ref}}(s \mid a)}$. Since our research is based on diffusion policies, we specifically define a chain advantage function for the diffusion policy to include all the latent variables in the middle. The marginalized form of the chain advantage function serves as the advantage function in our setting (Eq. 8). That means we need to consider the coupling of probability of the whole diffusion path rather than a state-action pair probability $\pi(a \mid s)$.

2. Difference with [2]. While we adopt some techniques from [2], such as Jensen’s inequality and the convexity of $-\log(\sigma)$ when deriving from Eq. 11 to Eq. 12 (we have accordingly cited [2] during the derivation), the problem setting, preference model, and derivation procedure are different. Additionally, we would like to highlight that the alignment of the diffusion policy via MLE is not our primary contribution. Instead, we primarily follow the approach developed for LLM in [2], and adapt it to the RL setting in our work.

   • Preference model and derivation procedure. [2] employs a reward-based preference model, which relies on the reward of the LLM's final output (see Eq. 3 in [2]) while regularizing the KL-divergence with respect to a reference policy. In contrast, this work adopts a regret (advantage)-based preference model (see Eq. 5) within the maximum entropy RL framework. Although both preference models ultimately lead to similar objectives for aligning the diffusion policy, the derivation procedures are somewhat different. Specifically, the derivation of this work is based on trajectory-wise alignment, whereas [2] only addresses the step-wise alignment. To accomplish this using the advantage-based preference model, we introduce the chain advantage function (i.e., Eq. 8), which is defined based on the chain reward function presented in [2]. We then calculate the expected value of the chain advantage function with respect to the diffusion latent variable, which serves as a key index for evaluating the trajectory.

3. Difference with [3]. While both works utilize the Beta prior with a MAP objective to model uncertainty during the alignment process, this work differs significantly regarding the problem formulation, underlying motivation, and approach to incorporating the Beta prior. Specifically,

   • Implementation. Specifically, [3] employs an iterative update rule, which is an approximation approach from numerical analysis (see Eq. 8 in [3]). This approach implicitly assumes an average reward of zero and allows for substantial fluctuation in the Beta prior parameters ($\alpha$, $\beta$). In contrast, our framework integrates the Beta prior directly into a trainable loss function for the target policy. We utilize a marginalized chain advantage function ($A(s,\overline{a})$) as our comparison metric, which is computed by $\mathbb{E}_{\pi\theta^*(a_t^{1:I}|a_t^0,s_t)}[A(s_t,\overline{a_t})]$. This distinction means the assumption of a zero-mean reward function is not applicable to our method. Furthermore, we have to carefully control the numerical magnitudes of $\alpha$ and $\beta$ to appropriately regularize the influence of the prior loss during training. These fundamental differences in implementation highlight a notable gap between the two approaches.

   • Approach. [3] adopts a two-step procedure, proposing a MAP objective for learning a distributional reward model. To achieve this, [3] introduces an iterative update rule that refines the reward model using the $\alpha$ and $\beta$ parameters output by the learned Beta prior model, until it converges to the maximum of its MAP objective. This reward model is then used for policy learning. In contrast, this work derives a unified MAP objective for directly aligning the diffusion policy in a single-step process, without the need for reward learning. We update our diffusion policy using a unified loss function (Eq. 17). By maximizing the likelihood of the diffusion policy's output under the learned Beta distribution, the process guides the policy to align the estimated $\phi(\tau)$ with their prior distribution $p_0(\phi(\tau))$. This approach is more straightforward and efficient.

[1] From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function. COLM 2024.
[2] Diffusion model alignment using direct preference optimization. CVPR 2024.
[3] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR 2025.

---

Q2. The experimental evaluation ....

Response. We appreciate your observation regarding the use of label flipping in our experimental evaluation of preference datasets. To address the inherent uncertainties characteristic of human preferences, we incorporated multiple noise models to simulate inconsistent human behavior. Specifically, our evaluation, as detailed in [1], includes the Stochastic Noise Model, the Myopic Noise Model, and the Irrational Noise Model, with the latter being equivalent to label flipping. The experiment results are provided in Table 4. We believe these diverse models collectively provide a robust framework for capturing the complexities and uncertainties inherent in human preferences.

[1] Robust preference-based reinforcement learning with noisy preferences. ICML 2024.

---

Q3. The necessity of the Beta distribution ... diverse human preferences.

Response. Thank you for your question. We would like to have a deeper discussion on Beta Distribution.

• Effectiveness of Beta Distribution. The Beta distribution proves highly effective for modeling human preferences in our framework due to its intuitive representation of confidence and uncertainty. It is the conjugate prior for the Bernoulli distribution on $\phi(\tau)$. Its two parameters, $\alpha$ and $\beta$, directly reflect accumulated "positive" and "negative" votes (e.g., preferred vs. unpreferred trajectory instances), allowing for a flexible update of beliefs as new preference data arrives (Lines 219–220). Other distributions are not naturally bounded on [0, 1] nor built for voting, which limits their suitability for proportion-based data. The Normal distribution’s unbounded support and symmetric shape, the Uniform’s fixed variance, and the Gamma’s semi-infinite range hinder their flexibility for diverse preference patterns.

• Unimodality of Beta Distribution. Since our study adheres to the standard setting of PbRL where preferences for a single sample are binary—either positive or negative, the aggregated win-probabilities remain effectively unimodal and are well-fit by a single Beta. For example, they either concentrate near 0 or 1 when consensus is strong, or near 0.5 when feedback is mixed. We acknowledge that truly multi-modal or list-wise preference patterns may necessitate more expressive priors such as mixtures of Beta distributions, which is a promising direction for future work.

---

Q4. In the introduction Line 54 ... in the main text.

Response. We apologize for the missing explanation. This process is very similar to choosing one of two candidates. Human annotators act as voters, casting preferences for trajectories (candidates) through pairwise comparisons in an election. The Diff-UAPA framework aggregates these "votes" using a MAP objective, incorporating a Beta prior to model uncertainty arising from diverse or conflicting preferences, akin to handling voter disagreements in a probabilistic voting system.

---

Q5. It appears that UA-PbRL ....

Response. Thanks for your question. In this paper, we compare UA-PbRL in our in-depth study (i.e., Table 4), evaluating its performance in MuJoCo environments under different types of noise. However, we acknowledge the importance of testing them in the main experimental evaluation, the results are:

This demonstrates that Diff-UAPA outperforms UA-PbRL, primarily owing to the powerful modeling capabilities of the diffusion policy.

---

Q6. While the paper ... which seems contradictory.

Response. Thank you for raising this point. The iterative preference alignment process in our framework is distinct from the offline or online RL setting. As outlined in Lines 126–134, iterative preference alignment refers to the method by which the preference dataset is constructed, labeled from various people, and utilized during training. This process does not specify the data collection mechanism, such as whether the agent interacts with the environment or relies on a pre-collected dataset. Thus, it is fully compatible with the offline training approach described in the paper, resolving any apparent contradiction.

Dear Reviewer, we greatly appreciate your constructive comments. We have seriously considered your suggestions, and we hope the following responses can address your concerns:

---

Q1. Training informative beta prior leads to additional computation costs.

Response. Thank you for raising this concern. As discussed in Appendix D.3, in this work, we use efficient techniques like the reparameterization trick to improve scalability. In practice, the computational cost of training the Beta model is similar to training a reward model in traditional PbRL. Since our method avoids training a reward model, the added cost is less effective compared to conventional PbRL. Additionally, the extra computational cost only slightly increases training time by a few minutes, while the subsequent RL phase is much more demanding, often taking several hours.

---

Q2. It seems that the baseline methods are not iteratively trained to adapt to evolving preference signals in the same manner as Diff-UAPA.

Response. Thank you for your insightful comment, and apologies for the confusion. As described in Line 260, our experiments consist of four rounds of iterative updates, each with a fixed number of episodes. This protocol is applied identically to all methods, including the baselines, so every algorithm is retrained on the newly acquired preference data at each round. We have clarified this procedure in the revised manuscript.

---

Q3. Can we come up with the proposed method's online counterpart?

Response. Thanks for your question. The proposed Diff-UAPA can be effectively generalized to an online setting due to the inherent flexibility of its core components, namely the Beta prior and MAP objective, which are not limited to offline datasets. The Beta prior’s ability to model uncertainty in human preferences makes it particularly suitable for handling dynamically evolving feedback from diverse user groups in real-time. To achieve this online counterpart, a feedback interface can be implemented to collect pairwise preference data (or from a pre-trained critic model) during or after trajectory execution in an interactive environment. This feedback can be stored in a dynamic preference replay buffer, which is incrementally updated as new trajectories are generated. The Beta prior parameters ($\alpha$, $\beta$) are then updated, enabling the diffusion policy to adapt continuously to new preferences while maintaining robustness to uncertainty.

---

Q4. Could the author provide a discussion on the advantages and disadvantages of the proposed method over simple robust methods such as data filtering and label smoothing?

Response. Thanks for your valuable question. We would like to provide a more detailed discussion as follows:

1. Advantages: UAPA offers significant advantages over data filtering and label smoothing. These Robust PbRL methods (e.g., label smoothing, data filtering) generally aim to exclude noisy or inconsistent data from the training process. While this may help reduce the impact of outliers, it also risks discarding valuable information if certain data points are mistakenly deemed as outliers. This filtering approach can result in lost opportunities for learning from diverse, potentially useful preferences. Besides, such techniques are usually static, lacking mechanisms to adjust to temporal or contextual changes in preference data. In contrast, UAPA employs a Beta prior to explicitly model the uncertainty inherent in diverse human preferences. Rather than eliminating uncertain data points, our framework assigns them lower confidence weights, allowing these samples to contribute conservatively to policy learning. This ensures that outliers or inconsistent preferences are not outright discarded but are instead integrated in a manner that minimizes their negative impact while preserving the richness of the preference dataset.

2. Disadvantage: Data filtering and label smoothing are computationally lightweight, as they involve simple rule-based preprocessing of the dataset. Training the Beta prior model in Diff-UAPA requires variational inference, which adds computational overhead compared to simpler methods. However, a small additional time cost (a few minutes) compared to the more demanding reinforcement learning phase is totally acceptable. (See more details in Appendix E)

We have included such a discussion in our revised paper to enhance clarity.

---

Q5. Could the author consider other iterative method baselines for a fair comparison with Diff-UAPA?

Response. Thank you for your question, and we apologize for any confusion. While none of our baseline methods include mechanisms to handle noisy preference labels, they all undergo four rounds of iterative updates during training, following the same procedure as Diff-UAPA (see Line 260). Therefore, these baseline methods are also iterative in nature.
To better address your concern, we additionally included an iterative preference alignment method, IPO [1]. The results are as follows:

We observe that Diff-UAPA consistently outperforms IPO, primarily due to its uncertainty-aware capabilities.

[1] IPO: Iterative Preference Optimization for Text-to-Video Generation. arXiv, 2025.

---

Dear Reviewer, we sincerely value your time and effort in evaluating our work. We have prepared comprehensive responses and clarifications to address each point you raised. We hope these responses can resolve your concerns.

---

Q1. The paper’s novelty is unclear, as the two key components seem to rely heavily on prior works: The diffusion policy alignment framework in Section 4.1 builds directly on Wallace et al. (2024), which introduces Direct Preference Optimization (DPO) for text-to-image diffusion models. The Beta prior model in Section 4.2 is inspired by Xu et al. (2025), which uses a Beta distribution for uncertainty-aware preference alignment in offline settings. While the combination of these components is novel, the paper fails to sufficiently demonstrate its unique innovations tailored to the problem at hand, weakening its claim to breakthrough innovation.

Response. Thank you for raising this concern. We would like to provide a more detailed discussion highlighting the differences between this paper and the two prior works as follows.

1. Difference with [1]. While we adopt some techniques from [1], such as Jensen’s inequality and the convexity of $-\log(\sigma)$ when deriving from Eq. 11 to Eq. 12 (we have accordingly cited [1] during the derivation), the problem setting, preference model, and derivation procedure are different. Additionally, we would like to highlight that the alignment of the diffusion policy via MLE is not our primary contribution. Instead, we primarily follow the approach developed for LLM in [1], and adapt it to the RL setting in our work. Specifically,

   • Problem setting. [1] is formulated in the context of LLM alignment, where rewards are assigned exclusively at the final step, and preferences are based solely on the final output of the LLM. This approach optimizes the LLM’s ultimate output, without considering intermediate steps or the overall process leading to the final result (see Eq. 14 in [1]). In contrast, Eq. 12 in this paper extends the framework to a trajectory-wise setting within the RL field. The key distinction is that in RL, we incorporate intermediate rewards, adjusted by a discount factor $\gamma$, which accounts for the cumulative discounted rewards throughout the trajectory. As a result, this paper formulates preference alignment based on the entire trajectory rather than focusing solely on the final state-action pair.

   • Preference model and derivation procedure. [1] employs a reward-based preference model, which relies on the reward of the LLM's final output (see Eq. 3 in [1]) while regularizing the KL-divergence with respect to a reference policy. In contrast, this work adopts a regret (advantage)-based preference model (see Eq. 5) within the maximum entropy RL framework. Although both preference models ultimately lead to similar objectives for aligning the diffusion policy, the derivation procedures are somewhat different. Specifically, the derivation of this work is based on trajectory-wise alignment, whereas [1] only addresses the step-wise alignment. To accomplish this using the regret (advantage)-based preference model, we introduce the chain advantage function (i.e., Eq. 8), which is defined based on the chain reward function presented in [1]. We then calculate the expected value of the chain advantage function with respect to the diffusion latent variable, which serves as a key index for evaluating the trajectory.

2. Difference with [2]. While both works utilize the Beta prior with a MAP objective to model uncertainty during the alignment process, this work differs significantly regarding the problem formulation, underlying motivation, and approach to incorporating the Beta prior. Specifically,

   • Problem formulation and motivation. In [2], the authors introduced an uncertainty-aware PbRL framework that specifically addresses epistemic uncertainty arising from an offline preference dataset with imbalanced comparison frequencies across different trajectories. In their setting, the fewer compared trajectories introduce greater epistemic uncertainty in reward prediction, due to the insufficient data available for these samples. Conversely, our work targets the aleatoric uncertainty associated with human preferences in an iterative updating dataset as defined in Definition 3.1. In this setting, different groups of human annotators may provide inconsistent or even conflicting preferences for the same pair of trajectories, which causes the aleatoric uncertainty that represents inherent stochasticity. In other words, by interpreting the parameters $\alpha$ and $\beta$ of the Beta distribution as representing the counts of 'vote' and 'unvote' human feedback, [1] uses a Beta prior to model the difference in the absolute values of $\alpha$ and $\beta$ across different trajectories (e.g., Beta(10,2) vs. Beta(100,20), where the former exhibits greater uncertainty due to fewer counts). In contrast, this work employs the Beta prior to model the relative strength of $\alpha$ and $\beta$ for different $\tau$ (e.g., Beta(6,6) vs. Beta(10,2), where the former exhibits greater uncertainty due to inconsistency of votes). (Part 1 of Section C)

   • Approach. [2] adopts a two-step procedure, proposing a MAP objective for learning a distributional reward model. To achieve this, [2] introduces an iterative update rule that refines the reward model using the $\alpha$ and $\beta$ parameters output by the learned Beta prior model, until it converges to the maximum of its MAP objective. This reward model is then used for policy learning. In contrast, this work derives a unified MAP objective for directly aligning the diffusion policy in a single-step process, without the need for reward learning. We update our diffusion policy using a unified loss function (Eq. 17). By maximizing the likelihood of the diffusion policy's output under the learned Beta distribution, the process guides the policy to align the estimated $\phi(\tau)$ with their prior distribution $p_0(\phi(\tau))$. This approach is more straightforward and efficient.

[1] Diffusion model alignment using direct preference optimization. CVPR, 2024.
[2] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR, 2025.

---

Q2. The theoretical contribution is primarily a reapplication of Beta priors (a well-known conjugate prior for Bernoulli distributions) to the diffusion policy context, without substantial new theoretical insights. The proof in Proposition 4.1, while correct, formalizes a standard Bayesian modeling approach rather than introducing a novel framework.

Response. Thank you for your insightful feedback. While both approaches indeed leverage the Beta prior within Bayesian framework, there are distinctions in how policies and Beta prior are updated.

• Iterative Alignment for Diffusion. Diff-UAPA introduces a novel iterative preference alignment framework tailored specifically for diffusion policies (Definition 3.1). Unlike traditional static offline preference alignment methods, our framework incrementally adapts the diffusion policy to preferences from diverse user groups, leveraging the sequential nature of diffusion models. This iterative process, supported by the Beta prior, enables dynamic updates that account for evolving preferences over time (Lines 284–287). In contrast, the approach in [1] operates in a static offline setting, lacking mechanisms for iterative adaptation to new preference data.

• Dynamic Adaptation of Beta Prior. Our proposed variant, Diff-UAPA-I, further distinguishes itself by enabling incremental training of the Beta prior, allowing it to adapt dynamically to changing preferences and environmental conditions. This dynamic adaptation is theoretically grounded in our iterative alignment framework, which ensures robust handling of evolving uncertainties in preference data. In contrast, the Beta prior in [1] is learned solely from offline demonstrations and does not incorporate mechanisms to adapt to new or evolving preferences, limiting its flexibility in dynamic settings.

[1] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR, 2025.

---