Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO

We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below:

---

Q1: Generalization on Other Autoregressive Architectures

Thanks for your advice! We agree that validating our findings on additional autoregressive architectures is crucial for strengthening the generalizability of our claims. To this end, we conducted additional experiments using Show-o, a distinct autoregressive image generation model. While Show-o adopts a discrete diffusion approach, it still generates visual tokens in an autoregressive fashion, though using a parallel decoding strategy that produces multiple tokens per step. Despite this difference, the underlying generation remains token-by-token and sequential in nature.

Due to time constraints, we focused on a representative subset of settings. The results consistently support our original findings:

1) DPO excels in in-domain tasks, while GRPO demonstrates stronger out-of-domain generalization.

2) The relative ranking of reward models on GenEval remains unchanged, with Unified Reward outperforming HPS under both RL algorithms.

These results confirm that our conclusions hold across architectures. We will include this extended analysis in the final version of the paper.

In-Domain Evaluation on T2I-CompBench

Out-of-Domain Generalization on GenEval

Impact of Scaling on In-Domain Proficiency

Impact of Scaling on Out-of-Domain Generalization

---

Q2: Clarification of In-Domain vs. Out-of-Domain Benchmark Choice.

Thank you for raising this important point. We agree that a clear justification is necessary, and we clarify our rationale below:

1) Prompt Complexity as the Core Source of Domain Shift.

Our in-domain/out-of-domain split is motivated primarily by the differences in prompt complexity and structure between the benchmarks, as outlined in Section 2.1. T2I-CompBench features long, compositional prompts designed to test fine-grained scene understanding and multi-attribute reasoning. In contrast, GenEval uses short, template-based prompts (e.g., “a photo of...”) aimed at assessing robustness to concise or minimal inputs. This creates a substantial domain gap in both linguistic form and generation difficulty. Supporting this, we observe that our baseline model (Janus-Pro) outperforms SD3-Medium on GenEval but underperforms on T2I-CompBench, indicating that proficiency on one benchmark does not transfer trivially to the other.

2) Additional Out-of-Domain Validation on DrawBench.

To further strengthen our claims and address the reviewer’s concern regarding potential attribute overlaps between benchmarks, we conducted supplementary experiments on DrawBench, a widely adopted benchmark for evaluating out-of-distribution generalization. The results remain consistent with our main findings: GRPO consistently outperforms DPO across multiple OOD-relevant categories, reinforcing its advantage in generalization.

We believe these points clarify the rationale behind our experimental design. We will incorporate this clarification and the new DrawBench results into the revised paper.

---

Q3: Expanding Visual Examples.

Thanks for your advice. It is necessary that including more visual examples can strengthen our claims. While the rebuttal format constraints prevent us from attaching new images or links, we will be sure to incorporate a more extensive set of visualizations in the camera-ready version of the paper.

---

Q4: Updating Citations to Peer-Reviewed Versions.

Thank you for the suggestion. We will carefully review all references and update any preprint citations to their corresponding peer-reviewed publications in the camera-ready version.

We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below:

---

Q1: Generalization on Other Autoregressive Architectures.

Thanks for your advice! We agree that validating our findings on additional autoregressive architectures is crucial for strengthening the generalizability of our claims. To this end, we conducted additional experiments using Show-o, a distinct autoregressive image generation model. While Show-o adopts a discrete diffusion approach, it still generates visual tokens in an autoregressive fashion, though using a parallel decoding strategy that produces multiple tokens per step. Despite this difference, the underlying generation remains token-by-token and sequential in nature.

Due to time constraints, we focused on a representative subset of settings. The results consistently support our original findings:

1) DPO excels in in-domain tasks, while GRPO demonstrates stronger out-of-domain generalization.

2) The relative ranking of reward models on GenEval remains unchanged, with Unified Reward outperforming HPS under both RL algorithms.

These results confirm that our conclusions hold across architectures. We will include this extended analysis in the final version of the paper.

In-Domain Evaluation on T2I-CompBench

Out-of-Domain Generalization on GenEval

Impact of Scaling on In-Domain Proficiency

Impact of Scaling on Out-of-Domain Generalization

---

Q2: Insufficient Statistical Rigor.

Thanks for your advice! To address this, we have rerun a representative set of our key experiments with three different random seeds.

The updated tables below report the mean and standard deviation. The results show that performance fluctuations are minimal, and our original conclusions remain well-supported. We will integrate these statistically-validated results into the final version of the paper.

In-Domain Evaluation on T2I-CompBench

Out-of-Domain Generalization on GenEval

Impact of Scaling on In-Domain Proficiency

---

Q3: Counterintuitive Overfitting of GRPO Compared to DPO with Data Scaling.

Thank you for the insightful observation. While this trend may appear counterintuitive at first, we believe it reflects GRPO’s stronger on-policy optimization, which enables the model (Janus-Pro) to reach its performance ceiling on GenEval more quickly.

As shown, GRPO peaks with the Double Size dataset; further scaling leads to overfitting to in-domain patterns, thereby degrading out-of-distribution performance. In contrast, DPO benefits more gradually from increased data, likely due to its off-policy nature and slower convergence, which delays performance saturation. We will clarify this interpretation in the revision.

---

Q4: Potential Typo Inquiry for DPO Value in Table 5.

Thanks for pointing out. We appreciate your careful review. This was indeed a typographical error. The DPO score in the Overall column for the Sampling 8 setting in Table 5 should be 79.23, not 55.17. We will correct this in the revised version.

---

Q5: Human Evaluation Comparing GRPO and DPO.

Thank you for the suggestion. We conducted a human evaluation comparing GRPO and DPO models trained with both HPS and Unified Reward. For each setting (in-domain and out-of-domain), 100 prompts were randomly sampled, and annotators compared image pairs, one from each method, choosing a preferred output or marking a tie.

These results from human evaluators align with our core findings: DPO performs better in-domain, while GRPO generalizes more effectively out-of-domain. We will include this evaluation in the revised version to strengthen the empirical support for our findings.

We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below:

---

Q1: Novel Insights Beyond Known Properties of GRPO and DPO in Autoregressive Image Generation.

Thanks for your thoughtful question. While some high-level traits of GRPO and DPO are known, our work presents the first systematic empirical study of these methods in the distinct setting of autoregressive text-to-image generation (T2I-AR), uncovering insights that do not directly follow from prior findings in text-based autoregressive tasks, due to the significant domain gaps between understanding and image generation settings.

1. Domain-Specific Challenges in T2I-AR: Unlike text generation, T2I-AR models operate over sequences of quantized visual tokens from VQ-VAE spaces, where rewards are derived from subjective and non-symbolic measures like aesthetics and semantic alignment. These differences fundamentally alter how RL algorithms behave. Prior assumptions, such as the robustness of critic-free methods, had not been tested in this setting. Our study fills this gap by empirically evaluating GRPO and DPO under these domain-specific constraints.

2. Distinct and Actionable Insights: Beyond confirming known trends, we provide new findings tailored to T2I-AR:

   1) We show that DPO's generalization is substantially more sensitive to the choice of reward model than GRPO's (Sec. 2.3, Table 2), a vulnerability less emphasized in textual domains.

   2) Crucially, we establish that the intrinsic generalization capability of a reward model correlates directly with the RL algorithm's out-of-domain performance (Sec. 2.3, Fig. 2a), suggesting a novel and practical path for improving generalization by focusing on better reward model design.

Together, these insights advance our understanding of how RL paradigms interact with the unique structure of visual token spaces, reward feedback, and training dynamics in T2I-AR, contributions not addressed in previous work on autoregressive generation.

---

Q2: Missing Mathematical Details for DPO and GRPO in Autoregressive Image Generation.

Thank you for raising this important point. In the revision, we will include the core training objectives for both GRPO and DPO in the context of autoregressive image generation. Specifically:

For GRPO, we will present the advantage normalization and KL-regularized optimization:

$A_i = \frac{r_i - \mathrm{mean}({r_i})}{\mathrm{std}({r_i})}$

$$$$

For DPO, we will formalize its preference-based loss without an explicit reward model:

$$L_{\text{DPO}} (\pi_\theta, \pi_{ref}) = -\mathbb{E} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(o_w|q)}{\pi_{ref}(o_w|q)} - \beta \log \frac{\pi_\theta(o_l|q)}{\pi_{ref}(o_l|q)} \right) \right]$$

These formulations reflect how both methods are applied to discrete token sequences in autoregressive image generation. We will incorporate them into Section 2.2 of the revised version, along with clear contextual explanations.

---

Q3: Concerns About Inconsistencies in Visual Evidence.

Thank you for pointing this out. We acknowledge that some visual examples may not have been sufficiently representative, and we appreciate your careful observation.

To address this, we will revise the figures in the final version by replacing inconsistent examples with clearer, more representative samples that align with our quantitative findings. We also plan to expand the set of comparative visualizations to more clearly support each key claim. We will ensure that the visual evidence in the paper accurately and convincingly reflects the empirical results.

---

Q4: Clarifying Table Presentation for Better Readability.

Thank you for the suggestion. To clarify, the Overall column in Tables 2, 3, and 5 already represents the average across all attributes, though this was not made explicit. In the revised version, we will clearly note this and bold the highest values in key columns to improve clarity and ease of comparison.

Thanks to the author for the detailed rebuttal. The author addressed most of my concerns, , and my revised score is therefore more positive.

However, there is still a small question: in actual image-to-text applications, is GRPO or DPO more suitable? From the author's experiments, it seems that DPO has a better result, as it performs better on the in-domain evaluation than GRPO, while the performance on out-of-domain evaluation is is close to GRPO.

We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below:

---

Q1: Limited Scope of Out-of-Distribution Generalization Evaluation.

Thank you for the insightful suggestion. In response, we have extended our evaluation to include DrawBench, a more challenging and compositional out-of-domain benchmark. Due to time constraints, we focused on representative experiments using two reward models. The results are as follows:

These findings are consistent with our original conclusions: GRPO consistently demonstrates stronger generalization than DPO, even on a more compositional and demanding benchmark. We believe this additional evidence strengthens the robustness of our claims, and we will include these results and a corresponding discussion in the final version.

---

Q2: Insufficient Statistical Rigor.

Thanks for your advice! To address this, we have re-executed a representative subset of our core experiments using three distinct random seeds. The revised tables below present the mean and standard deviation. The findings indicate that performance variations are negligible, and our initial conclusions remain strongly supported. We will incorporate these statistically-validated results into the final manuscript.

Table 1: In-Domain Evaluation on T2I-CompBench

Table 2: Out-of-Domain Generalization on GenEval

Table 3: Impact of Scaling on In-Domain Proficiency

---

Q3: Absence of PPO Baseline for Contextual Comparison.

Thank you for this valuable suggestion. While PPO is indeed a foundational RL algorithm, its application to autoregressive text-to-image generation remains a largely unexplored research direction, and its reliable implementation is impeded by significant technical constraints:

1) Intractable Value Estimation in Autoregressive Generation.

In PPO, a learned critic must accurately predict the final reward, comprising global aesthetic quality and text alignment, based on intermediate, partially generated image token sequences. This constitutes a highly unstable and noisy task, as confirmed by emerging work [1,2] in related domains.

2) Variance-Induced Instability and Training Collapse.

Due to the difficulty of accurate value prediction, the value function incurs high variance. This undermines advantage estimation in PPO, leading to unstable policy updates, a known failure mode in actor‑critic setups when the critic is mis-trained [3,4].

By contrast, preference‑based methods like DPO and GRPO bypass the need for a critic entirely, relying instead on pairwise comparisons or group‑based policy optimization. These methods have been shown to provide greater stability and effectiveness in multimodal and autoregressive generation tasks, including recent applications in diffusion‑based image generation and multimodal CoT reasoning.

On this basis, we focused our evaluation on DPO and GRPO. That said, we recognize the potential value of a simplified PPO baseline for contextual reference and will consider including such an experiment in future extensions of this work.

[1] A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning.

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

[3] On Proximal Policy Optimization's Heavy-tailed Gradients.

[4] No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO.

---

Q4: Potential of GRPO Warm-up Followed by DPO Refinement.

Thanks for the insightful suggestion. To evaluate this hybrid strategy, we conducted a new experiment where GRPO was first applied to half of the T2I-CompBench training set, followed by DPO fine-tuning on the remaining half using the GRPO-trained checkpoint for initialization.

The results confirm that this sequential approach produces a balanced trade-off: it improves DPO’s generalization and enhances GRPO’s in-domain performance, but does not surpass either method on their respective strengths. We will include this observation and its implications in the final version.

---

Q5: Consistency of Findings Across Janus 1B and 7B Variants.

Thanks for the suggestion. To assess the generalizability of our findings, we conducted a representative subset of experiments using the Janus 1B variant due to time constraints. Encouragingly, the results remain highly consistent with our main observations:

1) DPO retains its advantage on in-domain tasks, while GRPO continues to generalize better out-of-domain.

2) The ranking of reward models on GenEval is preserved, with Unified Reward outperforming HPS under both RL algorithms, as previously reported.

These findings reinforce the robustness of our conclusions across model scales. We will include these additional results and analyses in the final version.

In-Domain Evaluation on T2I-CompBench

Out-of-Domain Generalization on GenEval

Impact of Scaling on In-Domain Proficiency