MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning

Q1: Comparison with the performance of MORvS

MODULI consistently outperforms the strong MORvS baseline, leading in 20/24 metrics across 12 Complete datasets(Tab. 4) and surpassing the baseline in 59/72 metrics on Shattered & Narrow datasets(Tab. 5 & 6).

Q2: Why use 3 seeds?

We collect 501 sample points for each seed to calculate the metrics, fully covering the preference space, which results in small variances. At the same time, we aligned with the setting of D4MORL benchmark, it also uses 3 seeds. We conducted a large number of performance and ablation experiments(2 levels * 3 types * 6 tasks), demonstrating the reliability of the MODULI.

Q3: Significantly extend the Pareto front

Due to task constraints, the Pareto front in Fig.3 can only expand by a small margin. Compared to other methods, MODULI demonstrates stronger generalization and an "expansion" phenomenon. To avoid misunderstanding, we replace "significantly" → "slightly" in revision. Additionally, extensive experiments and visualization(fig.7) proves that MODULI can enhance generalization.

Q4: Confusing expressions, incomplete issue and typos

We apologise for the unclear description in the introduction. We will provide a brief introduction to the concept in the Introduction and explain them in detail in the Method. And we have carefully reviewed your suggestions part. Thank you for pointing them out! We will thoroughly review the paper, check for grammatical issues and verify the references in revision.

Q5: Intuition for Equation 11

We trained a diffusion model $p$ to generate trajectories corresponding to ID preferences. Then, we attempt to learn the pattern of change (for example, when the preference shifts from energy efficiency to high speed, the amplitude of the Swimmer's movements gradually increases). Therefore, our goal is to learn an adapter $p^*$, which applies this pattern to OOD preferences to achieve better generalization. In Equ. 11, the exponential term represents increasing the likelihood of preference $c+$ and decreasing the likelihood of preference $c−$. η Represents the guidance strength weight in classifier-free guidance for the diffusion model. Based on Equ. 11, we can use an adapter-style method to adjust the original noise prediction model ($p$). Now, the noise prediction combines the noise from the original model and the adapter.

We apologize again for the confusing expressions and undefined symbols. We will provide definitions and include thorough intuitive explanations.

Q6: Detailed limitations

We assume a linear preference space, such as $w_1 + w_2 = 1$, which is the standard practice in most MORL papers. When the linear assumption does not hold, our core method remains applicable, but additional representations are required. And very sorry for the confusion. We will change the “structured state space” to “low-dimensional state space” (Distinguished from image input). “Continuous action” should be “continuous action space”, distinguished from discrete action space. We believe that MODULI can support tasks with these new modalities with minimal additional adaptation (e.g., encoder). We left it for future work.

Q7: Loss-weight trick

Sorry for the confusion, we will include detailed implementations in the appendix. The diffusion planner generates a state sequence $x^0(\tau) = \left[ s_0, \cdots, s_{H-1} \right]$. Since closed-loop control is used, only $s_0$ and $s_1$ are most relevant to the current decision. When calculating the loss, the weight of $s_1$ is increased from 1 to 10, which is considered more important.

Q8: Compute Equation 7

In D4MORL dataset, each trajectory contains a target preference $\omega$, and $g$ is the return of the trajectory. Since $D_p$ consists of trajectories corresponding to Pareto front points, it represents the optimal level of policy in the dataset. Therefore, for each $(g, \omega)$ sampled from $D_p$, $g$ is the maximum return corresponding to $\omega$. Then, a predictor $R(\omega)$ for the maximum possible return can be obtained with Equ.7. It can be seen as a fit to the Pareto front in the dataset.

Q9: RD metric

Please note that we trained a predictor for each task (e.g., ant-expert) to evaluate RD, rather than one for each algorithm. The detailed process: we fixed the Ant-expert-complete dataset to trained $R_\psi$. Since expert and complete trajectories were used, we can estimate the optimal $R$ for each sampled preference $w$, these ground truth information is only used for evaluation. Then, for Ant-expert-{complete/narrow/shattered}, for the trajectories sampled from OOD preference points, RD is defined as the difference between the ground truth return from $R_\psi$ and the actual obtained return. The results are averaged over all OOD preference points. A smaller RD indicates that the algorithm can better align with the target preference when the target preference is OOD, thus demonstrating better generalization ability.

...

# Create a loss_weight and assign a higher loss weight to the next observation (o_1).
next_obs_loss_weight = 10
loss_weight = torch.ones((batch_size, horizon, obs_dim)) 
loss_weight[:, 1, :] = next_obs_loss_weight  

def diffusion_loss_pseudocode(x0, condition):
    """
    Pseudocode function to demonstrate the loss function in diffusion models
    
    Variable descriptions:
    x0: Original noise-free data - [batch_size, horizon, obs_dim]
    condition: Conditional information - [batch_size, condition_dim]
    xt: Noisy data after noise addition - same shape as x0
    t: Time points in the diffusion process step - [batch_size]
    eps: Noise added to x0 - same shape as x0
    loss_weight: Loss weight coefficient - scalar or tensor matching x0's shape
    """
    
    # Step 1: Add noise to the original data
    xt, t, eps = add_noise(x0)  # Generate noisy xt from x0
    
    # Step 2: Process conditional information
    condition = condition_encoder(condition)  # Encode the original condition information
    
    # Step 3: Predict noise through the diffusion model
    predicted_eps = diffusion_model(xt, t, condition)  # Model predicts the added noise
    
    # Step 4: Calculate mean squared error
    loss = (predicted_eps - eps) ** 2  # Mean squared error between predicted noise and actual noise
    
    # Step 5: Apply loss weights
    # We assign a higher weight to the prediction of the s_1
    loss = loss * loss_weight  
    
    # Step 6: Calculate average loss
    final_loss = loss.mean()  # Calculate the average loss
    
    return final_loss

We thank the reviewer for the insightful and useful feedback, please see the following for our response.

Q1: Comparison of Parameters for Different Baseline Algorithms

Thank you for your suggestion! We would like to clarify that in our experiments, we standardized the parameter size of all baseline algorithms to the same magnitude as much as possible. A complete list is provided in the table below:

We believe your suggestion is very correct, and we will include this comparison in the revised version.

Q2: Novelty of MODULI

We would like to clarify that MODULI is not merely an application of diffusion models in offline RL. The novelty of MODULI lies in the following three aspects:

• A new research problem: When only incomplete offline data (narrow/shattered) is available, the preference for generalization to OOD is crucial for multi-objective RL. For the first time, We model Offline MORL from the perspective of generative models and introduce diffusion models into offline MORL. We found that their expressive and generalization capabilities are highly effective for MORL problems. Relying solely on Decision Diffuser[1] cannot achieve the performance of MODULI, the following two techniques are important.
• Improved return normalization methods: Traditional return normalization guided by the only maximum reward has critical flaws in scenarios with conflicting multi-objectives. We designed two return normalization methods (NPN/PPN) tailored for multi-objective problems, which are crucial for diffusion models in MORL.
• Better OOD Preference Generalization: Through the study of multi-objective task datasets, we found that as preferences change, policies in multi-objective tasks will show certain changing trends, suggesting a potential direction for addressing the challenge of policy generalization in multi-objective optimization tasks. Inspired by prior work, we introduced a sliding adapter to enhance generalization capability, achieving significant results while many implementation details differed from [2], including the fine-tuning method, the calculating unit strength and the final integration of the outputs from two models. We focus on stronger generalization ability rather than precise conceptual control over new attributes.

[1] Ajay A et al. Is conditional generative modeling all you need for decision-making. ICLR2023.

[2] Gandikota et al. Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models. ECCV2024.

We thank the reviewer for the insightful and useful feedback, please see the following for our response.

Q1: Novelty of MODULI

We would like to clarify that MODULI is not merely an application of diffusion models in offline RL. The novelty of MODULI lies in the following three aspects:

• A new research problem: When only incomplete offline data (narrow/shattered) is available, the preference for generalization to OOD is crucial for multi-objective RL. For the first time, We model Offline MORL from the perspective of generative models and introduce diffusion models into offline MORL. We found that their expressive and generalization capabilities are highly effective for MORL problems.
• Improved return normalization methods: Traditional return normalization guided by the only maximum reward has critical flaws in scenarios with conflicting multi-objectives. We designed two return normalization methods (NPN/PPN) tailored for multi-objective problems, which are crucial for diffusion models in MORL.
• Better OOD Preference Generalization: Through the study of multi-objective task datasets, we found that as preferences change, policies in multi-objective tasks will show certain changing trends, suggesting a potential direction for addressing the challenge of policy generalization in multi-objective optimization tasks. Inspired by prior work, we introduced a sliding adapter to enhance generalization capability, achieving significant results while many implementation details differed from [1], including the fine-tuning method, the calculating unit strength, and the final integration of the outputs from two models. We focus on stronger generalization ability rather than precise conceptual control over new attributes.

Q2: The impact of the degree of conflict between multiple objectives on the MODULI performance

We conducted a comprehensive evaluation on datasets of varying quality (expert, amateur) and datasets with different levels of OOD (complete, narrow, shattered), all of which have varying degrees of conflicting objectives, such as speed and energy. As shown in Fig. 3 (page 7), we were surprised to find that when the dataset is incomplete, baseline algorithms like BC fail completely under OOD preferences due to conflicts, and RvS tends to fit a single solution regardless of different preferences. In contrast, MODULI adapts to varying levels of conflict and dataset quality, demonstrating excellent performance.

Q3: Can you comment on how your method would perform when the number of objective increases?

• Our experiments include the hopper-3obj dataset, which involves 3 conflicting objectives. It is worth noting that as the number of objectives increases, the policy for given preference becomes more complex. MODULI demonstrates outstanding performance in the three-objective task, achieving over a 20% improvement compared to the strongest baseline MORvS (Table 1).
• From the perspective of method design, MODULI does not assume any specific number of objectives. The proposed new normalization methods and sliding adapter can handle scenarios with more objectives.

[1] Gandikota et al. Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models