Flow-GRPO: Training Flow Matching Models via Online RL

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows. Due to space limits, part of the responses to [W1 is included in other reviewers' rebuttals. We apologize for the inconvenience and kindly refer the reviewer to those sections for details.

[W1] Comparison with other RL methods

Thanks for your suggestions. First, in Appendix C.2, we already provide the comparison with other RL baselines, including offline SFT/DPO/RWR and their online variants. Results show that Flow-GRPO trains more stably than other RL baselines and ultimately attains the highest evaluation reward. In addition, SPO is a very insightful work, but it requires a step-aware preference model to label win-lose pairs. In contrast, our method only needs a reward signal from the final, clean image. The final-image equivalent of SPO is DPO, which we have already compared against. We will cite SPO in our revision and clarify this distinction.

Beyond the existing results, we have also conducted additional comparative experiments, these results will be included in the revision. Due to space limitations, we kindly refer the reviewer to Reviewer JifQ's ［W3］and [Q1 & Q2] for further details. We will include this information in the revised version of the main paper. We summarised below:

1. Comparison with other RL baselines on GenEval.

Please see Reviewer JifQ's ［W3］

2. Comparison with DDPO on PickScore.

Please see Reviewer JifQ's ［W3］

3. Comparison with ORW on CLIP Score.

Please see Reviewer JifQ's [Q1 & Q2]

[W2] Whether improvements in specific metrics come at the cost of degradation in others

In Table 2, we have already covered the other critical aspects you mentioned. Specifically, it reports image quality metrics such as Aesthetic and DeQA, along with human preference metrics like ImageReward, PickScore, and UnifiedReward. Flow-GRPO improves the task-specific metrics without degrading these additioanl metrics.

[W3] The alleviation of reward hacking stems primarily from GRPO itself.

We emphasize that we do not claim to "invent" the KL constraint, instead our results show that a carefully tuned KL penalty can match the high rewards of a KL-free run while preserving image quality, albeit at the cost of slightly longer training. Importantly, the KL term is not viewed as a necessary component of GRPO, previous studies [1] have even reported improved scores after removing it. Our contribution is that, through comprehensive ablation studies, we demonstrate that the KL constraint is critical to the Flow‑GRPO algorithm, enabling the reward to keep rising without any collapse in fidelity or diversity. This is not a general consensus in online RL for generative models, we believe our finding offers useful empirical guidance to the community.

[1]. DAPO: An Open-Source LLM Reinforcement Learning System at Scale.

[W4] The effectiveness on other Flow-based models/diffusion-based models

Thank you for this valuable suggestion. We have run Flow-GRPO on FLUX.1-Dev using PickScore as the reward signal. The reward curve rises steadily throughout the training without noticeable reward hacking. We report:

(1) The reward value over the training process

(2) Comparison between FLUX.1-Dev and FLUX.1-Dev + Flow-GRPO on DrawBench.

Although our experiment mainly focuses on flow-matching models, Flow-GRPO could be applied directly in diffusion models without modification. If a diffusion model were used instead, this stochasticity is already built in their DDPM-style sampling, so the same pipeline in Flow-GRPO can be directly utilized in diffusion models' optimization.

[W5] Flow-GRPO performs worse than the original SD3.5-M on texture generation.

In T2I‑CompBench++, the “texture generation” evaluates the ability to render material appearance (e.g., plastic, metallic, and wooden). This axis is essentially orthogonal to the objectives we optimise in GenEval, which focus on counting, color binding, and 2D spatial relations, and our training prompts contain no vocabulary or descriptions of object material. Given that the model receives no reward signal in that direction, a slight decline from 0.7338 to 0.7236 is unsurprising and negligible when set against the broader gains Flow‑GRPO achieves on its target tasks.

[W6] The technical contributions are relatively incremental

Thank you for the comment. Our paper does not aim to propose a novel RL algorithm but instead aims to address a critical open question: Can contemporary generative models be improved through online RL, and how to achieve this for the flow-matching backbones now dominating SOTA generation models. Before our work, PPO/GRPO have shown remarkable success in understanding tasks with LLMs, while their applicability to flow-matching generation models has not yet been demonstrated. We show that with carefully chosen modifications, flow-matching models can profit from online RL, yielding measurable gains in multi generation tasks. Our results provide key design guidelines and empirical evidence for applying online RL to flow-based generators, opening new directions for future research in this area. We summarise our key contributions as follows:

• We establish a well-defined MDP for flow-matching generators. GRPO presupposes stochastic, step-by-step transitions so that a policy $p_{\theta}(a_t \mid s_t)$ is well defined, yet a standard FM model evolves deterministically under its ODE formulation. By leveraging the connections between FM models and score-based generative models, we convert the ODE into an SDE with controlled noise supplying the required randomness. This injects exactly the stochasticity needed to instantiate the MDP without altering the model's marginal distribution.
• We introduce several practical strategies that make online RL both efficient and robust. We find that online RL does not require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process with 4x speedup. Additionally, we show that the KL constraint could suppress reward hacking, reaching the high reward while preserving image quality and diversity.
• We provide empirical validation of the effectiveness of Flow-GRPO. We evaluate Flow-GRPO on T2I tasks with various reward types. This includes: 1) verifiable rewards for compositional generation and visual text rendering; 2) model-based rewards for human-preference alignment. Across all settings, Flow-GRPO yields consistent improvements, demonstrating that the proposed framework is a promising path for enhancing SOTA generative models.

[Q1] Why use different random noise? The authors should provide an additional experiment comparing different initial noise + ODE sampling paths versus the current approach

We initialize each rollout with different random noise to increase exploratory diversity during RL training. We perform an additioanl ablation to confirm this claim. With SD-3.5-M as the base model and PickScore as the reward, we compare Flow-GRPO with different initial noise against Flow-GRPO with the same initial noise. The variant with different noise consistently achieved high rewards during the training process. The detailed results are reported below:

A comparison based on “different noise + pure ODE sampling” is inapplicable. GRPO requires a stochastic, step‑by‑step transition model so that the policy $p_{\theta}(a_t \mid s_t)$ is well defined. ODE sampling is deterministic, therefore it does not furnish an MDP; under that setting the GRPO objective cannot be optimised.

[Q2] Prompt ratio justification

The prompt ratio was selected empirically according to the relative difficulty of each task. The base SD-3.5-M model handled single-object and two-object scenes almost flawlessly, yet struggled with position, counting, and attribute binding. We therefore allocated a larger portion of training prompts to the harder categories and a smaller portion to the easier ones. This trick does not affect our contributions and conclusions, as each RL baseline is trained with the identical backbone and the same training prompts.

Thank you for the authors' efforts in addressing the concerns raised in my initial review.

[W1]

The authors should conduct fair comparisons with ReFL using differentiable reward functions (e.g., HPSv2 or aesthetic models), otherwise it is difficult to demonstrate that GRPO outperforms existing ReFL algorithms in terms of performance.

[W2, W3]

I appreciate the authors' clarification and efforts made in this regard. This discussion has effectively addressed my concerns. However, I still recommend that the authors clearly indicate in Table 2 which metrics are used for training versus evaluation, and update future versions with the details mentioned in W1 (data update frequencies of different methods) as well as a clearer description of reward hacking.

[W4]

The authors' response in this section only demonstrates the advantages of "GRPO" over "PPO/DPO" but fails to explain why "GRPO" is more suitable for "Diffusion models" compared to existing reinforcement learning algorithms (such as ReFL). Furthermore, while ReFL requires differentiable reward functions, it does not need multiple rollouts like GRPO, making it difficult to intuitively understand where GRPO's advantages lie.

The authors have resolved 2 of my main concerns in this response, therefore I am raising my score to Borderline Reject for now. However, I still have questions regarding W1 and W4, and hope the authors can actively respond to these remaining concerns.

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.

[W1] The paper claimed to be the "first method" to incorporate online RL into flow-based models, which I find to be an overclaim.

We appreciate the reviewer’s comment and agree that it's important to clarify what we mean by “online reinforcement learning” in this context. In our work, we specifically refer to algorithms from the typical reinforcement learning literature, such as PPO [1], GRPO [2], DQN [3], DDPG [4], and SAC [5].

This categorization is consistent with the distinctions made in prior work. For example, the Diffusion-DPO [6] paper explicitly classifies DPOK [7] and DDPO [8] as RL-based methods, while grouping other approaches into separate categories. Furthermore, the original DPO [9] paper characterizes its own contribution as “a simple RL-free algorithm for training language models,” reinforcing the distinction between RL-based and non-RL-based alignment methods.

We will revise the manuscript and acknowledge prior work such as Diffusion-DPO [6], ORW [10], RAFT [11], and DRaFT [12], and discuss how they relate to our setting.

[1] Proximal policy optimization algorithms
[2] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
[3] Playing Atari with Deep Reinforcement Learning
[4] Continuous control with deep reinforcement learning
[5] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
[6] Diffusion Model Alignment Using Direct Preference Optimization
[7] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models
[8] Training Diffusion Models with Reinforcement Learning
[9] Direct Preference Optimization: Your Language Model is Secretly a Reward Model
[10] Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization
[11] Raft: Reward ranked finetuning for generative foundation model alignment
[12] Directly Fine-Tuning Diffusion Models on Differentiable Rewards

[W2] The contributions seem limited to me.

Thank you for the comment. Our paper does not aim to propose a novel RL algorithm but instead aims to address a critical open question: Can contemporary generative models be improved through online RL, and how to achieve this for the flow-matching backbones now dominating SOTA generation models. Before our work, PPO/GRPO have shown remarkable success on understanding tasks with LLMs, while their applicability to flow-matching generation models have yet not been demonstrated. We show that with carefully chosen modifications, flow-matching models can profit from online RL, yielding measurable gains in multi generation tasks. Our results provide key design guidelines and empirical evidence for applying online RL to flow-based generators, opening new directions for future research in this area. We summarise our key contributions as follows:

• We establish a well-defined MDP for flow-matching generators. GRPO presupposes stochestic, step-by-step transitions so that a policy $p_{\theta}(a_t \mid s_t)$ is well defined, yet a standard FM model evolves deterministically under its ODE formulation. By leveraging the connections between FM models and score-based generative models, we convert the ODE into an SDE with conytolled noise supplying the required randomness. This injects exactly the stochasticity needed to instantiate the MDP without altering the model's marginal distribution.
• We introduce several practical strategies that make online RL both efficient and robust. We find that online RL does not require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process with 4x speed up. Additionally, we show that the KL constraint could suppress reward hacking, reaching the high reward while preserving image quality and diversity.
• We provide empirical validation of the effectiveness of Flow-GRPO. We evaluate Flow-GRPO on T2I tasks with various reward types. This includes: 1) verifiable rewards for compositional generation and visual text rendering; 2) model-based rewards for human-preference alignment. Across all settings, Flow-GRPO yields consistent improvements, demonstrating that the proposed framewrok is a promising path for enhancing SOTA generative models.

[W3] Only a limited comparison with the RL baselines was provided in the appendix on a single metric. As this paper was not for the benchmark track, such an organization might be misleading for the RL community

Thank you for pointing out this issue! We agree that a broader evaluation is necessary. We have therefore added some extensive experiments that will be included in the revision. These experiments are summarised below:

1. Comparison with other RL baselines on GenEval.

Similar to the Figure 6 in the appendix, we compare Flow-GPRO against offline SFT/DPO and their online variants. For each method we track the GenEval evaluation reward and apply early stopping when visual quality collapse is observed. The reward scores, measured at increasing number of training prompts, are list as below:

We observe that offline baselines like offline DPO and offline SFT hit a ceiling early, with the reward converging at roughly 0.76. Their online variants improve faster but are fragile, which emerges severe reward-hacking and ultimately causes the models to collapse. By contrast, our Flow-GRPO trains smoothly, pushing the reward up to about 0.95 without incuring significant hacking issues.

Note that the reviewer mentioned RAFT: for each input $x_t$, it generates a group of $G$ responses and selects the one with the highest reward $y_t$ within the group for supervised fine-tuning. In fact, this is essentially the same as our baseline labeled "SFT" in the paper.

2. Comparison with DDPO on pickscore

DDPO was originally designed for diffusion backbones, so we adapted it to flow-matching models by applying our ODE-to-SDE conversion. Using SD-3.5-M as the base model and PickScore as the reward signal, we track the evaluation reward over the entire training process, as shown below:

We observe that DDPO's reward rises more slowly than Flow-GRPO's and collapses in the late stages, whereas Flow-GRPO trains steadily and continues to improve throughout the training process.

[Q1 & Q2] Results for CLIP Score and ORW

Thank you for the suggestion. Since the official ORW repository only provides a partial implementation, we re-implemented ORW based on their repo within our reward-weighted regression baseline. The loss function is as follows:

fm_loss = ((vt - ut) ** 2).mean(dim=(1, 2, 3))
w2_loss = ((vt - vt_ref) ** 2).mean(dim=(1, 2, 3))
total_loss = torch.mean(torch.exp(beta * rewards) * fm_loss + alpha * w2_loss)

Experimental setup:

• Training prompts: Same as those used in our Human Preference Alignment task.
• ORW hyperparameters: β = 0.5, α = 1 (lower values led to unstable training); steps_per_epoch (i.e., how frequently the data-collecting policy is updated) was selected from {20, 40, 100, 400} based on best performance.

Reward scores on the test set over training steps:

Following ORW’s Table 1, we randomly sampled 50 DrawBench prompts and generated 64 images per prompt to compute CLIP and Diversity scores. We used ORW’s best-performing checkpoint on the test set (at step 480). Flow-GRPO outperforms ORW on both metrics:

We will include these results in the revision.

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.

[W1] About the originality;

Thank you for the comment. Our paper does not aim to propose a novel RL algorithm but instead aims to address a critical open question: Can contemporary generative models be improved through online RL, and how to achieve this for the flow-matching backbones now dominating SOTA generation models. Before our work, PPO/GRPO have shown remarkable success in understanding tasks with LLMs, while their applicability to flow-matching generation models has not yet been demonstrated. We show that with carefully chosen modifications, flow-matching models can profit from online RL, yielding measurable gains in multi generation tasks. Our results provide key design guidelines and empirical evidence for applying online RL to flow-based generators, opening new directions for future research in this area. We summarise our key contributions as follows:

• We establish a well-defined MDP for flow-matching generators. GRPO presupposes stochastic, step-by-step transitions so that a policy $p_{\theta}(a_t \mid s_t)$ is well defined, yet a standard FM model evolves deterministically under its ODE formulation. By leveraging the connections between FM models and score-based generative models, we convert the ODE into an SDE with controlled noise supplying the required randomness. This injects exactly the stochasticity needed to instantiate the MDP without altering the model's marginal distribution.
• We introduce several practical strategies that make online RL both efficient and robust. We find that online RL does not require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process with 4x speedup. Additionally, we show that the KL constraint could suppress reward hacking, reaching the high reward while preserving image quality and diversity.
• We provide empirical validation of the effectiveness of Flow-GRPO. We evaluate Flow-GRPO on T2I tasks with various reward types. This includes: 1) verifiable rewards for compositional generation and visual text rendering; 2) model-based rewards for human-preference alignment. Across all settings, Flow-GRPO yields consistent improvements, demonstrating that the proposed framework is a promising path for enhancing SOTA generative models.

[Q1-1] Why choose flow matching as the base model?

We choose flow matching models because it provides faster sampling speed at comparable quality. Contemporary leading image generation models (e.g., SD3 [1], FLUX [2]) and video generation models (e.g., Sora [3], Wan2.1 [4], Seedance [5]) now adopt flow matching in place of traditional diffusion, which means our method could apply directly to their post‑training stage.

[1]. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.
[2]. FLUX.
[3]. Sora: Video generation models as world simulators.
[4]. Wan: Open and Advanced Large-Scale Video Generative Models.
[5]. Seedance 1.0: Exploring the Boundaries of Video Generation Models.

[Q1-2] How would the approach differ if a diffusion model were used

Thanks for asking this clarification question!

1. GRPO requires an MDP with stochastic step-to-step transitions so that a policy $p_{\theta}(a_t \mid s_t)$ is well defined. A flow-matching model, in its standard ODE form, is purely deterministic, and there is no randomness in a single denoising step. To address it, we therefore convert the ODE to an equivalent SDE to inject randomness, creating the stochastic transitions needed to instantiate the MDP.
2. Our proposed method could be applied directly in diffusion models without modification. If a diffusion model were used instead, this stochasticity is already built in their DDPM-style sampling, so the same pipeline in Flow-GRPO can be directly utilized in diffusion models' optimization.

[Q2] The design principles behind the reward function f defined on line 119?

Thank you for the question. There seems to be a misunderstanding: in our notation, f refers to the GRPO objective function, not the reward model (which is denoted by r). A proper reward model typically needs to assign a scalar score to each generated sample (higher means better), and does not need to be differentiable. The three reward functions we adopt in the manuscript are defined on Line 168, Line 169, and Line 176.

The GRPO objective (f) is designed to ensure stable and monotonic policy improvement through three key principles:

1. Trust Region. The core of the function is a clipping mechanism that uses min and clip functions to limit the magnitude of policy updates. This creates a trust region, allowing only small, safe adjustments around the current policy. This effectively prevents performance collapse due to overly large update steps and ensures training stability.

2. Advantage-Weighted Updates. Our goal is to maximize the objective f, which corresponds to maximizing $r_t Â_t$. Therefore, the direction of updates is guided by the advantage function $Â_t$:

• When $Â_t > 0$ (i.e., the action is better than average), the objective reduces to maximizing $r_t$. Since the denominator of $r_t$ has no gradient, this effectively means increasing the numerator — that is, increasing the probability of that action.
• When $Â_t < 0$ (i.e., the action is worse than average), the probability of that action is decreased.

3. KL Divergence Regularization. The loss function includes a KL divergence penalty term, $-β D_{KL}(π_θ || π_{ref})$. It acts as a soft constraint, preventing the new policy $π_θ$ from deviating too far from a known reference policy $π_{ref}$ (often the pretrained policy).

[Q3] What are the main factors limiting the scalability of the proposed method?

Thank you for raising this point. We discuss two main factors that govern scalability.

1. Efficiency. Scaling the post-training stage requires a more efficient training strategy. Our manuscript introduces a denoising reduction strategy that trains the model with only a quarter of the normal sampling steps, achieving roughly a 4x speed-up with very subtle loss in quality. Further works could attain gains by adopting a dynamic data selection, for instance, periodically discarding samples whose advantage estimates show low variance.
2. Avoid Reward Hacking. Reward hacking would become increasingly problematic as the training scale grows. We observe that a KL regulariser can curb the hacking effectively, allowing more optimisation steps before the model starts to hack the reward function (see Figure 8). Even so, an ultimate safeguard is a robust and accurate reward model, which makes it harder for the generator to discover loopholes. Further work on powerful, ensemble-based rewards will therefore be one key to scaling the algorithm.

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.

[W1] The conversion from Flow ODE to SDE is not a novel contribution, as stochastic sampling for flow-based models has already been explored in prior works.

Thank you for raising this concern. Our ODE-to-SDE conversion is not intended as a standalone theoretical contribution, but as a practical tool that equips Flow-GRPO with the stochastic transitions required for a valid MDP. Applying GRPO-style optimization in FM models demands a sampling procedure whose step-by-step randomness does not disturb the model's marginal distribution. We therefore construct an SDE that is distribution-equivalent to the original ODE, which is largely inspired by the prior flow-matching theory [1] and shares a similar formulation with prior works such as [2]. The contribution lies not in the mathematics of the conversion itself, but in adapting and repurposing it to make online RL feasible and empirically effective for contemporary flow-matching backbones.

[1]. Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control. ICLR 2025.
[2]. SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers. ECCV 2024.

[W2-1] This work appears to be largely an application of existing techniques to diffusion models, which limits its novelty.

Thank you for the comment. Our paper does not aim to propose a novel RL algorithm but instead aims to address a critical open question: Can contemporary generative models be improved through online RL, and how to achieve this for the flow-matching backbones now dominating SOTA generation models. Before our work, PPO/GRPO have shown remarkable success in understanding tasks with LLMs, while their applicability to flow-matching generation models has not yet been demonstrated. We show that with carefully chosen modifications, flow-matching models can profit from online RL, yielding measurable gains in multi generation tasks. Our results provide key design guidelines and empirical evidence for applying online RL to flow-based generators, opening new directions for future research in this area. We summarise our key contributions as follows:

• We establish a well-defined MDP for flow-matching generators. GRPO presupposes stochastic, step-by-step transitions so that a policy $p_{\theta}(a_t \mid s_t)$ is well defined, yet a standard FM model evolves deterministically under its ODE formulation. By leveraging the connections between FM models and score-based generative models, we convert the ODE into an SDE with controlled noise supplying the required randomness. This injects exactly the stochasticity needed to instantiate the MDP without altering the model's marginal distribution.
• We introduce several practical strategies that make online RL both efficient and robust. We find that online RL does not require the standard long timesteps for training sample collection. By using fewer denoising steps during training and retaining the original steps during testing, we can significantly accelerate the training process with 4x speedup. Additionally, we show that the KL constraint could suppress reward hacking, reaching the high reward while preserving image quality and diversity.
• We provide empirical validation of the effectiveness of Flow-GRPO. We evaluate Flow-GRPO on T2I tasks with various reward types. This includes: 1) verifiable rewards for compositional generation and visual text rendering; 2) model-based rewards for human-preference alignment. Across all settings, Flow-GRPO yields consistent improvements, demonstrating that the proposed framework is a promising path for enhancing SOTA generative models.

[W2-2] How does the exploration-exploitation tradeoff in the Flow-GRPO compare with diffusion models or standard RL settings?

Thank you for the question. In Flow-GRPO, the exploration–exploitation tradeoff is governed by the noise scale $\sigma_t$ in Equation (9). Conceptually, $\sigma_t$ plays a role similar to the $\epsilon$ in $\epsilon$-greedy strategies in standard RL settings: a larger $\sigma_t$ increases exploration by introducing greater stochasticity into the denoising process, but at the cost of reduced policy quality and potential instability or suboptimal behaviors.

We provide an empirical analysis of this tradeoff in the paragraph starting at line 225, where we systematically vary $\sigma_t$ and study its effect on final performance. The results demonstrate a clear tradeoff, highlighting the importance of tuning $\sigma_t$ appropriately to balance exploration and exploitation.

[W3] The ablation on the group size G on GRPO.

Thank you for this valuable suggestion. We have added an ablation study on the group size $G$ using PickScore as the reward function. When we reduced the group size to $G=12$ and $G=6$, training became unstable and collapsed at step 778 and step 742, respectively. In contrast, $G=24$ remained stable throughout the full process. We logged the PickScore on the test set before and after the collapse, as shown below.

Our conclusion is that a small group size leads to inaccurate advantage estimates, inflating the variance and eventually triggering collapse. A similar phenomenon is also reported in [3, 4]. We will add these results in the revision.

[3] ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. Arxiv 2025.
[4] AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning. Arxiv 2025.

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.

[Q1] How does this ODE-to-SDE conversion affect metrics like FID/SSIM/ human evaluations?

We evaluate the impact of converting ODE sampling to SDE sampling using SD‑3.5‑M as the base model. We compared 40-step ODE and 40-step SDE sampling on MSCOCO‑2014 for FID/IS/CLIP Score, and on DrawBench for aesthetic/DEQA/ImageReward/PickScore/UnifiedReward. The two settings yield essentially the same performance at 40 steps, which aligns with our derivation in Sec. A showing that both share the same marginal distribution. For clarity, SDE sampling is only used during training; all reported metrics are computed from 40-step ODE samples. Consequently, the ODE-to-SDE conversion has no material impact on FID/ID or human-preference metrics in our experiments.

[Q2-1] Why does using fewer steps improve reward?

Thank you for the question. We view denoising reduction as a strategy to boost efficiency, not to raise the ultimate reward ceiling. In addition to Figure 5, Section C.2 in the Appendix extended ablations on visual text rendering and human-preference alignment. Across these three tasks, a 10-step schedule converges to similar final rewards compared to the 40-step baseline while cutting wall-clock training time by roughly 4x. The higher reward observed therefore reflects faster convergence, not a higher performance.

[Q2-2] The number of updates in the graph

In Figure 5, we measure progress by GPU‑hours, i.e., wall‑clock cost. If we re‑index the curves by the number of optimizer updates, the 10‑step and 40‑step schedules are nearly equivalent. One difference is that each update in the 40-step schedule requires four times the accumulation of the 10‑step schedule's, owing to its 4x longer denoising steps.

[Q3] How strong is the generalization? In Table 4, the authors show that the model generalizes well for objects > 4. When does it break?

Thank you for the suggestion. Following the protocol of Table 4, we extend the generalization test and require the model to generate images with 8/10/12 objects, an extremely challenging setting in which the base SD-3.5-M model almost never succeeds. Flow-GRPO still achieves a substantial improvement, although the absolute success rate naturally decreases as the task becomes more difficult. The results are summarised below:

[Q4] Will the model be robust to long visual text as well? When does the model break?

Thank you for the suggestion. Our text-rendering checkpoint was trained on short phrases (2-4 words), so we further evaluate its generalization on prompts containing a full 10-word sentence. For illustration, a representative long‑text prompt is:

The final, unplaced piece of a giant, intricate jigsaw puzzle, held in a hand above the puzzle. The piece itself has the words, “The final piece that makes the entire picture make perfect sense.

In this harder setting the base SD-3.5-M model degrades sharply in accuracy, whereas Flow-GRPO maintains most of its performance. The results are summarised below:

We sincerely thank Reviewer 2Psq for your thoughtful and detailed review. We are very pleased that you recognize that our work shows online RL can improve media generation significantly, and that you found our paper has enough details for proper reproduction and is very useful to the community.

Your comments regarding FID/SSIM, performance with fewer steps, and generalization have been instrumental in helping us refine our work. To that end, we have provided detailed responses in our rebuttal, including additional experiments specifically designed to address your concerns.

With the discussion phase drawing to a close, we would be grateful if you could let us know whether our clarifications and new results have resolved your initial concerns. We greatly appreciate your time and would welcome any further feedback.