Boosting Offline Multi-Objective Reinforcement Learning via Preference Conditioned Diffusion Models

Although DIFFMORL shows some performance improvement over baselines on the D4MORL benchmark, I regret to say that this work offers limited insights and insufficient contributions to the community. In my understanding, DIFFMORL is fundamentally a standard diffusion planning algorithm. Apart from the design of the conditions, it does not differ significantly from decision diffuser (DD). Thus, the key focus of the discussion lies in the new components of DIFFMORL: independent preference encoding (IPE) and mixup-based training (MT).

• IPE is not an innovative design; rather, it is a fairly natural one. In most applications of diffusion models, generating conditions are independently encoded and incorporated into the diffusion network. Within RL community, the idea of encoding different conditions to adjust decision-making objectives has already been discussed in prior work like DD. While IPE is not novel, a more in-depth analysis by the authors—showing exactly how IPE and PC affect trajectory generation and ultimately influence performance—could still be a valuable contribution. Unfortunately, the paper only demonstrates that IPE results in higher scores than PC on MO-HalfCheetah.

• I find the use of MT confusing. Unless I have misunderstood the paper, the authors appear to perform simple linear interpolation between two batches of data to generate synthetic samples. While it is reasonable to apply linear interpolation to preferences or rewards, I seriously question whether interpolating between trajectories yields meaningful data—more likely, it would generate incoherent or nonsensical behavior. For example, if one trajectory corresponds to a HalfCheetah running at 2 m/s and another at 10 m/s, a linear interpolation with a 0.5 weight would not produce a 6 m/s run, but rather an erratic and malfunctioning control signal. Furthermore, the paper lacks an in-depth analysis of MT, making it difficult to trust its effectiveness. I also remain uncertain about whether this method can be broadly applicable, as it seems to provide benefits only in DIFFMORL based on the current experiments.

Overall, my main concern is the novelty of the methodology used in this work.

1. To me, the paper is basically a combination of the Diffuser and DT methods and mix-up techniques and applies them to solve MORL tasks. I don't see any theoretical challenges posed by combining these methods and applying them in MORL besides implementation and experimentation. The application is still meaningful, though, but this is somewhat below the bar of top conferences like ICLR.

2. I have concerns about the mixup-based data augmentation methods. It basically linearly interpolates reward, preference weight, as well as state and action information in the dataset. The first two are fine. However, for state and action data, I really don't think it makes much sense to linearly add them up together, especially is we want to generate actual trajectory rather than conducting classification. If this technique is so useful, we should do this when training text2image models. I'm sure the generated images won't make any sense if we do this in pretraining.

3. When referring to CFG technique in Line 166, you should cite the original CFG paper rather than song's.

• Limited novelty. At least for me, using diffusion models to improve data modeling ability in reinforcement learning is not new. It has been proposed and validated by many papers previously. Additionally, the paper augments offline datasets by data mixup, which is also not an originally proposed trick. Previous papers [1,2] have demonstrated that we can augment offline datasets via diffusion synthesis, and thus, the policy performance can be improved. It is better to discuss this technique and explain why you do not consider using it.
• Experimental results are sufficient but not strong enough. In Table 1, the proposed method performs almost similarly to the previous diffusion baseline MODULI.

Reference

[1] Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning, NeurIPS 2023

[2] Synthetic Experience Replay, NeurIPS 2023