Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective

Thanks for your all feedbacks and nice suggestions! I'm glad you appreciated our work.

Here are some responses to your questions:

---

'I have some concerns about the novelty of GMPO. Despite its simplicity, I believe this is somewhat very similar to the AWR algorithm, just that it optimizes a diffusion policy instead of a Gaussian policy. Also, this algorithm has already been verified in the image generation field.'

It’s true that GMPO bears a structural similarity to AWR in a narrow sense, as the latter uses the following form:

$\mathcal{L}\_{\mathrm{AWR}}(x)=\exp{A(x)}\log{p(x)}$

In fact, any generative model that can be trained with a maximum likelihood objective, $\mathcal{L}\_{\mathrm{MLE}}(x)$, to maximize its log-likelihood (without requiring a strict bound), can be applied with the AWR framework in a general sense:

$\mathcal{L}\_{\mathrm{AWR}}(x)=\exp{A(x)}\mathcal{L}\_{\mathrm{MLE}}(x)$

For GMPO, the $\mathcal{L}\_{\mathrm{MLE}}(x)$ corresponds to the flow matching loss or score matching loss. Both of these methods have been shown to serve as evidence lower bounds (ELBOs) for maximum likelihood estimation in diffusion or flow models. This has been formally proven in the following works:

• "Maximum likelihood training of score-based diffusion models"
• "Maximum likelihood training for score-based diffusion odes by high order denoising score matching"
• "Improved techniques for maximum likelihood estimation for diffusion odes".

In this sense, GMPO can be seen as a generalization of AWR, as it extends the framework to diffusion and flow models.

---

'When applying GMPG, as I understand, Neural ODE needs to be leveraged in order to estimate logp of \mu. I'm concerned about the computation efficiency of this method. Could you compare it with SRPO and Diffusion-QL?'

We can compare with SRPO based on our implementation.

For the finetuning stage of halfcheetah-medium-expert-v2-SRPO, it reaches its best performance after approximately 1.8 hours on a single A100 GPU. For the finetuning stage of halfcheetah-medium-expert-v2-GMPG, it reaches its best performance after approximately 1.7 hours on a single A100 GPU.

The time may vary across datasets and tasks, but the computational cost of GMPG is not significantly different from SRPO in practice.

---

'Also, I think for GMPG, your policy model has to be a Gaussian but cannot be a diffusion policy. For Diffusion QL, your policy can be a Diffusion policy. Is this correct?'

This is not entirely correct.

For GMPG, the pretrained policy can be either a diffusion model or a flow model, obtained via score matching or flow matching, as long as $\log{\pi}$ is analytical and can be computed. After finetuning, the policy is no longer the original pretrained diffusion or flow model—it becomes a flow model with a modified generation trajectory. Importantly, $\log{\pi}$ remains analytical for the finetuned policy.

For Diffusion-QL, the policy is a diffusion model with score matching constraints in the original paper. However, it can be extended to a flow model with flow matching constraints as well.

For both GMPG and Diffusion-QL, the policy can degenerate into a Gaussian model, which is equivalent to the DDPG algorithm in a general sense.

Thanks for your feedback and all suggestions! I'm glad you appreciated our work.

Here are some answers to your questions:

---

'Are the authors proposing a "framework" or merely organizing "experimental code"?' 'How well does the flexibility of the GenerativeRL framework extend to the compatibility with other reinforcement learning algorithms?'

GenerativeRL is a framework designed for research at the intersection of generative models and reinforcement learning (RL). It goes beyond just organizing experimental code, offering key features that distinguish it as a framework:

Key Features of GenerativeRL:

Modularity: GenerativeRL is highly modular, allowing users to swap or combine different components, such as generative models or RL algorithms. This flexibility is demonstrated across the experiments in the paper and further detailed in the appendix.

Extensibility: The framework provides a clear API for adding or modifying components. Users can customize neural network architectures, loss functions, and training procedures while reusing the provided generative models and RL algorithms. For instance, one can easily integrate a new diffusion model by modifying only the diffusion process coefficients, leaving the rest unchanged. Additionally, the generative models and RL algorithms can be used independently, allowing users to integrate their own datasets or environments.

Compatibility: GenerativeRL was designed with long-term maintenance and development in mind. It defines flexible data shapes and types, as detailed in the Framework section. Additionally, it supports automatic differentiation, which is crucial for deep learning, and is compatible with distributed training using either native PyTorch DDP or libraries like Accelerate.

Minimal Dependencies: We’ve taken a minimalistic approach to dependencies, focusing on clarity and simplicity.

Unittests: The codebase includes extensive unittests to ensure the correctness of the implementation. These tests cover the core components of the framework, such as generative models and RL modules, running automatically during continuous integration.

Documentation: The codebase is well-documented, with detailed explanations of how components interact. It includes guides on implementing generative models and RL algorithms, with predefined data shapes, class types, and function augmentations. We provide Jupyter notebook examples and toy demos to help users get started quickly.

Open-source: GenerativeRL is designed for open-source, allowing the community to contribute and extend the framework. We welcome feedback, bug reports, and pull requests to improve the framework and make it more accessible to researchers and practitioners via PYPI.

We apologize that we cannot provide more information in the paper and supplementary materials due to review anonymity rules, but we have provided most of the ready-made code that constitutes this open source library in the supplementary materials.

We are committed to maintaining and improving GenerativeRL by integrating new research and community feedback. We hope that the framework will serve as a valuable tool for researchers and practitioners working on generative models and RL.

---

'Does the paper "classify existing works into two categories" or "survey prior works, identify their limitations, and propose two new algorithms as a solution"?'

The paper does both: it classifies existing works into two categories and surveys prior works, identifying their limitations and proposing two simplified algorithms as a solution.

By comparing existing methods, we aim to provide a comprehensive overview of the state-of-the-art in generative policies. Rather than introducing new techniques in GMPO or GMPG, we simplify existing approaches into two forms. This simplification is motivated by our observation that many methods share similar training objectives, despite being implemented differently.

We categorize these methods based on two core objectives:

• Maximizing an advantage-weighted log-likelihood.
• Maximizing a guided function sampled from the generative model, with proximal constraints.

From a foundational perspective, we argue that these two objectives capture the essence of current methods, with the rest being implementation details. Many existing methods are overly complex and obscure which components are crucial for performance. For instance, techniques like action tensor clipping in works such as Diffusion-QL are rarely discussed explicitly.

Additionally, most existing methods lack modularity, making it difficult to swap components for ablation studies or comparison.

Our experiments demonstrate that the proposed algorithms are competitive with existing methods, while being simpler and less technically decorated.

---

'Does this study focus specifically on offline RL, or is it broadly applicable to general RL?'

The experiments in this paper focus on offline RL, but the proposed methods, GMPO and GMPG, are general and can be applied to online RL as well.

We conducted some preliminary experiments using a two-stage training process: first, we pretrained the generative policy with offline datasets, then transitioned to online RL, where new data was collected and used to fine-tune both the policy and value function. These experiments showed that the online fine-tuned policy outperformed the offline pretrained one, demonstrating the methods' competitiveness in online RL. Similar experimental results can be found in Appendix F of the aforementioned work Diffusion-QL's paper, where they also found that using online data would further improve model performance.

However, since our focus in this paper is on revisiting generative policy optimization in offline RL, we leave a more systematic exploration of online RL for future work. In general, online RL poses additional challenges, such as balancing exploration and exploitation, which are beyond the scope of this paper.

---

'The connection between prior research and the present work is not sufficiently emphasized. It would also be helpful to introduce abbreviations (e.g., SfBC) in the Related Works section and explain the paper’s categorization in that context.'

Thank you for the suggestion. We will revise the paper to better emphasize the connection between our work and prior research, including the use of abbreviations in the Related Works section.

We already provide a brief overview of these connections in Appendix C, but we will ensure this information is included in the main text for clarity. We apologize for omitting it due to page limitations.

---

'What does "Any" in Table 1 signify? If it only refers to DDPM and VPSDE, does that mean methods based on flow models are not covered? Or, are flow models also included under "Any"?'

The term "Any" in Table 1 refers to the inclusion of all generative modeling methods, encompassing both diffusion models (such as DDPM and VPSDE) and flow-based models. Flow models are also considered under the "Any" category. To clarify this further, we will update Table 1 with additional details to avoid any potential misunderstandings.

---

'Were there no prior algorithms using flow models, or were they intentionally excluded? In the introduction, it was mentioned that offline RL focuses on diffusion models—are there no flow-based methods in this domain? (line 44, Table 1)'

Flow-based methods do exist in the domain of reinforcement learning (RL), and there is growing interest in using flow models for decision-making tasks. Since the introduction of flow matching for generative models in late 2022, over 500 papers have cited this approach, with more than 90 papers mentioning reinforcement learning (RL). Some of these works either introduce new RL algorithms or adapt flow-based methods to RL settings.

Here are several notable examples of flow-based methods in RL and decision-making:

• "Preference Alignment with Flow Matching": This work explores preference-based reinforcement learning (PbRL) using flow matching, similar to diffusion-based approaches but with flow models.
• "RF-POLICY: Rectified Flows are Computation-Adaptive Decision Makers": This paper introduces an imitation learning algorithm that utilizes rectified flows for adaptive decision-making.
• "Guided Flows for Generative Modeling and Decision Making": This work presents guided flows for generative modeling, with experiments demonstrating their application in reinforcement learning.
• "Riemannian Flow Matching Policy for Robot Motion Learning": A method focused on robot motion learning, integrating flow-based generative policies with Riemannian manifold constraints.
• "Affordance-based Robot Manipulation with Flow Matching": This paper applies flow-based models to robot manipulation tasks, driven by text-prompted affordance learning.
• "Generalized Schrödinger Bridge Matching": A generalized approach for bridging distribution matching in reinforcement learning, leveraging a flow-based framework.
• "π0: A Vision-Language-Action Flow Model for General Robot Control": This work applies flow models in vision-language-action tasks for robot control, demonstrating the versatility of flow-based policies.

Although these papers reflect the growing use of flow models in RL, they were not included in our current comparisons, mainly due to the particular focus of our work is different from these papers.

We will consider adding these references to provide a more comprehensive overview.

Dear authors, thanks for the rebuttal. I have carefully read your response and some of concerns are resolved. Currently, I have a follow-up question:

• In the revised manuscript, I have not able to clearly identify a formal derivation establishing theoretical connection between two objectives: "maximizing an advantage-weighted log-likelihood" and "maximizing a guided function sampled from the generative model". Could you elaborate or kindly point this connection in a rigorous way?

Thanks for your feedback and suggestions! I'm glad you appreciated our work.

Here are some responses to your questions:

---

'Are any new insights or empirical findings that emerged from their systematic analysis that go beyond prior work?'

From a theoretical perspective, our main contribution is the introduction of two new, simplified training schemes for generative models. By classifying prior methods into two categories, we show that these earlier approaches are special cases of our proposed methods using different training and inference schemes.

Empirically, our methods perform comparably or even better than previous approaches on several benchmark datasets and are more efficient to train. This highlights the effectiveness of both our methods and the components of prior techniques. Additionally, through thorough empirical studies, we demonstrate that several biases identified in prior research can be alleviated with our approach.

Overall, we provide a unified and standardized framework for training generative models in RL tasks, which we hope will serve as a valuable reference for future research.

---

'The performance of GMPO and GMPG seems similar to previous methods in locomotion tasks (table 3) and weaker than baselines in antmaze tasks (table 11, several baselines > 75, while GMPO 73.8 and GMPG 66.5). Why does this phenomenon exist, and is there some specific analysis about it? For example, are there any significant characteristics in locomotion tasks and antmaze tasks that result in the proposed algorithm, especially GMPG, performing poorly in the antmaze tasks?'

Our experiments were not optimized for peak performance on individual tasks but to demonstrate the broad applicability of our methods. We conducted a rough hyper-parameter search using a coarse grid (e.g., geometric sequences with multiples of 2) and selected configurations with generally strong performance. Unlike other works, we did not perform detailed hyper-parameter tuning for each task, which may explain suboptimal results in specific cases. However, overall, our methods' performance falls within the variance observed across these tasks, as we will elaborate in the following analysis.

The antmaze tasks are particularly challenging because the agent must navigate through mazes without getting stuck or falling over. In contrast, locomotion tasks are simpler, typically requiring only forward movement (e.g., running or jumping) without obstacles. In antmaze, if the agent flips upside down, it cannot recover, resulting in a zero reward for the episode. In locomotion tasks, the agent can still earn continuous rewards even with suboptimal actions, making its performance less sensitive to individual mistakes.

Moreover, the reward structure in antmaze is binary—the agent either reaches the goal and receives a reward or fails and gets zero. This makes performance highly dependent on avoiding mistakes over long episodes (1000 steps), especially in larger mazes. For instance, an agent that quickly reaches the goal but fails at a corner near the end would still receive zero reward, whereas a slower agent that completes the maze in 1000 steps would receive full reward.

This binary reward structure can feel counterintuitive. If I were the judge, I might award a higher score to the agent that reaches the goal faster, even if it fails near the end, as it demonstrates learning beyond simple imitation of expert behavior. So I suggest we should not take these grades too serious, not making it the only metric.

As discussed in Appendix D.2, Table 11 shows that in smaller mazes, all methods perform similarly. However, in larger mazes, reverse-KL methods like GMPG and diffusion-QL degrade more significantly, while forward-KL methods like GMPO and QGPO remain more robust. This difference arises from the distinct characteristics of forward-KL and reverse-KL approaches. Forward-KL methods act as filters, selecting high-value actions from the existing policy. In contrast, reverse-KL methods encourage more radical exploration, sometimes pushing the policy toward unreliable or unsafe regions. This divergence is also visible in Figure 1, where forward-KL methods stay closer to the pretrained model, while reverse-KL methods deviate more rapidly.

Thanks for your feedback and suggestions! I'm glad you appreciated our work, which I hope could be useful for your research.

Here are some responses to your questions:

---

'One of the my major concerns is the position of the paper. It is a bit confused that authors want to build a unified benchmark suite for evaluating various generative policies for RL or propose an algorithm that can be easily applied to train generative policies for RL. For the former, it will be better to include some details on experimental setup and training & evaluation procedure. For the latter, more analysis or stronger experimental results should support the author's claim.'

Thank you for this observation. We can understand your concern about the paper's positioning.

The motivation behind GenerativeRL is the inconsistency in how diffusion policies are defined and implemented across the field. Different training schemes, evaluation metrics, and neural architectures make it difficult to compare generative policies objectively. Our contribution, GMPO and GMPG, aims to simplify these variables and retain only the essential components. This allows for a more direct comparison of generative policies.

Our intention, therefore, is not to compete with existing policies or prove superiority but to provide a reliable baseline for future research. The experiments show that our approach performs well in RL tasks, and we believe this can help reset comparisons on a more level foundation. By maintaining this framework, we hope to support future work in this rapidly evolving area.

We hope this clarification can help you better understand our work.

---

'While the paper focus on offline RL algorithms, there are several works which utilize generative policies for online reinforcement learning [1, 2]. It might be better to cover those algorithms or the title should be revised to Offline RL.'

Thank you for these references. We will evaluate them and consider including them into the framework for future comparisons for online RL. We are also exploring online RL with generative models and have observed promising results in offline pretraining followed by online finetuning. A more comprehensive analysis will be provided in a future paper.

---

'In D4RL benchmark, Diffusion-QL achieves state-of-the-art performance even though other baselines are more recently proposed. Could authors validate why such phenomena happens?'

As shown in Table 3, Diffusion-QL performs similarly to other baselines with high-quality medium-expert data. The difference arises with medium and medium-replay data, where Diffusion-QL outperforms others. This suggests that Diffusion-QL is better at learning from lower-quality data.

We observe similar behavior in GMPG, which uses a reverse-KL objective like Diffusion-QL. This objective tends to explore more aggressively, sometimes pushing the policy towards risky regions, which can yield better performance when learning from imperfect data. By contrast, GMPO, QGPO, and IDQL use a forward-KL objective, which acts more conservatively and struggles with lower-quality data.

Diffusion-QL also incorporates several specific design choices—some mentioned in their paper and others in their code—which may contribute to its superior performance:

• Joint looping training: Policy and Q-function are trained simultaneously, akin to a moving average, which may stabilize training. While GMPG trains Q-function first via IQL, then finetuning policy.
• Value function training: They use a pure argmax Bellman backup, unlike GMPG's IQL training or QGPO's softmax-based approach.
• Action clipping: Diffusion-QL uses action clipping through out the inference from the diffusion policy if the $x_t$ is out of the action space $[-1, 1]$. This introduces a strong human-made prior by clipping actions to the valid range, effectively regularizing the policy. As the Q function is trained with the action sampled from the diffusion policy, it is regularized as well.
• Inference Enhancing: They generate multiple actions during inference and select the highest-value action, which boosts performance slightly.

GMPG does not use such tricks in implementation for providing a pure, simple and naive diffusion policy from the first principle.

---

'In figure 1, what is the optimal policy in this plot exactly? Does it indicate optimal policy introduced in Eq (5)? Is it tractable?'

In Figure 1, the star represents the best-performing policy we obtained during GMPO and GMPG training. It is not the exact optimal policy from Eq (5), but a model checkpoint saved during training. Therefore, it is tractable and represents the best policy we observed empirically.

Dear Reviewer,

Thank you for your feedback and for pointing out the error parameter in our demo Colab.

We have corrected the configuration issue in the Colab notebook GMPO.

We truly appreciate your valuable contribution to GenerativeRL.

Here are some responses to your further questions:

---

"However, I think those tricks should not be ignored, and if those tricks lead to significant improvement, we should take it."

We completely agree with your perspective. From an engineering or deployment standpoint, we should adopt every possible trick to maximize the model's performance.

However, for academic exploration, as emphasized in our paper titled "Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective", our primary objective is to investigate the performance improvements that can be achieved by using generative policies with as few additional tricks as possible. By doing so, we aim to conduct an empirical study of pure generative policies, rather than diffusion policies enhanced by other techniques. This approach addresses an important gap in prior works.

For example, in Diffusion-QL, neither action clipping nor inference enhancing tricks were explicitly discussed in their paper. This omission might confuse future researchers and could misleadingly attribute the contributions of action clipping and inference enhancing to the diffusion policy itself. This would be problematic, as these tricks are universal and applicable regardless of whether the policy is generative or not.

If other works, such as QGPO, do not apply such tricks, it would not represent a fair ablation to directly compare the performance of QGPO and Diffusion-QL. Our work aims to provide an intermediate and unified comparison of these algorithms in a fairer and more transparent manner.

We hope this addresses your concerns. Thank you again for your insightful comments.

Sorry for my late response. While I strongly agree that different training schemes, evaluation metrics, and neural architectures for diffusion policies hinders fair comparison, several tricks suggested by prior works in implementation level is also quite important. For example, authors claim that GMPO and GMPG achieves competitive performance compared to Diffusion-QL without several tricks such as joint training, generate multiple actions and choose one action, etc. However, I think those tricks should not be ignored and if those tricks lead to significant improvement, we should take it.

I appreciated authors to provide a comprehensive review of prior diffusion policies as it is hard to tracki so many papers which are proposed simultaneously nowdays. However, I'm skeptical on future researchers start their research upon this foundations rather than relying on several tricks already proposed in prior papers. Therefore, I keep maintaining my score.

+) I also briefly check the colab notebook demo in the GenerativeRL code and find that it does not work... Could authors check whether there is no bug in the current code?