Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

Weakness1: While the method is empirically effective, the authors offer limited theoretical analysis or formal justification for why the proposed multimodal GRPO and reward design are particularly suited to interleaved generation.

A: We thank the reviewer for pointing out the lack of theoretical justification. To address this concern, we provide conceptual justifications from two theoretical perspectives to support its design:

1. Unified Policy Optimization: In the main paper, we have introduced a unified policy optimization formulation in Eq. 3 and 4:

This formulation allows actions from the visual modality to be seamlessly integrated into the overall generation trajectory and jointly optimized alongside textual actions. By treating all modality transitions within a unified policy space, our approach enables global optimization over the entire multimodal sequence. This not only ensures semantic consistency across modalities but also facilitates effective gradient propagation through the interleaved structure, which is critical for learning coherent cross-modal dependencies.

2. Process-Level Reward Design: Previous works focusing on single-modality generation, such as Deepseek-R1, have explored introducing process-level rewards to better align generation with human preferences. However, they encountered a critical limitation: it is inherently difficult to accurately define and annotate what constitutes a "correct step" in complex reasoning or generation processes. This ambiguity in step boundaries leads to reward diffusion, where local credit assignment becomes unclear, resulting in noisy gradient updates and unstable learning dynamics. In contrast, interleaved multimodal generation naturally provides discrete and semantically grounded boundaries through modality switching (e.g., from text to image and back). These modality transitions serve as anchors for defining process-level steps, enabling more accurate and interpretable credit assignment during optimization. To facilitate understanding, we derive the optimization objective as:

When step-level definitions are incorrect or inconsistent, the resulting reward distribution becomes noisy and ill-posed, leading to unstable learning signals and suboptimal policy updates. This leads to high variance in gradient estimation. In our interleaved multimodal setting, modality transitions provide explicit segmentation of the trajectory, allowing us to define rewards locally per modality segment. This reduces ambiguity in credit assignment and improves sample efficiency during optimization.

We will include a more detailed theoretical discussion in the revised version to support this rationale.

Weakness2 & Q1: The process-level reward modeling and the need to generate multiple candidate sequences per prompt in GRPO introduce substantial computational costs, which may limit scalability. Can you provide more quantitative analysis on the computational and memory overhead introduced by the GRPO stage (e.g., compared to standard RLHF or SFT)? Please also discuss potential strategies for reducing this overhead, such as more efficient candidate sampling or reward estimation.

A: We appreciate the reviewer’s concern regarding the potential computational and memory overhead introduced by our GRPO stage.

1. We first would like to clarify that our process-level reward does not introduce additional computational burden—it simply redistributes the original outcome-level reward across different modality-specific steps. Therefore, it incurs negligible additional cost compared to standard RLHF reward computation. Furthermore, our group-wise sampling reduces training overhead compared to standard RLHF methods such as PPO, as it eliminates the need for training a separate critic model, which is often the most computationally intensive component in PPO-based pipelines.

2. To further support our claim, we provide a quantitative comparison of the computational and memory overhead introduced by GRPO relative to standard RLHF and supervised fine-tuning (SFT) in the table below. For fair comparison, the number of samples per iteration is fixed to 4. The results show that GRPO introduces only marginal overhead compared to SFT, and is significantly more efficient than PPO-style RLHF in both compute and memory. We will include this quantitative analysis in the revision. |Method|GPU Hours (per 1K steps)|Peak Memory (GB)| |:----|:----|:----| |SFT|9.7|30| |PPO-based RLHF|15.3|58| |Our GRPO|11.1|42|

3. To further reduce cost, we are actively exploring more efficient group-wise candidate sampling strategies (e.g., top-k filtering or nucleus sampling) and lighter reward estimation techniques that approximate process-level rewards without relying on full model evaluations at each step.

Weakness3 & Q2: Could you provide more details or experiments on the robustness and generalization ability of these reward models, especially when applied to out-of-distribution data or tasks beyond the current benchmarks? If not already included, consider testing or discussing how the method would adapt if reward models are imperfect or biased.

A:

1. To examine the robustness of reward models, we have included experiments in the main paper (Table 6) that specifically examine imperfect or biased reward models. In this setting, we replace the original reward model with CLIP, which serves as a weaker and less aligned alternative. The results show that while there is a slight performance drop (-2.56 on MMIE and -0.55 on InterleavedBench), our GRPO framework remains stable and continues to outperform baseline methods, highlighting its resilience to noise and bias in the reward signal.

2. To further address this concern, we conduct additional experiments following the MVoT protocol [1], where a suite of out-of-distribution, maze-specific tasks is used to evaluate the model’s visual reasoning capabilities across multimodal contexts. As shown in the table below, without any changes to the reward models, our method achieves substantial performance gains on these challenging benchmarks. This demonstrates the robustness and generalization ability of our reward design beyond the current benchmark tasks. |Method|MAZE|MINIBEHAVIOR|FROZEN-LAKE| |:----|:----|:----|:----| |MVoT|0.9295|0.9514|0.856| |Vila-U|0.7046|0.6527|0.5839| |Ours|0.9593|0.9706|0.8742|

Weakness4: Tables 3, 4, 5, and 6 are poorly presented with inconsistent sizes and font styles, making the paper appear unpolished and disorganized.

A: We thank the reviewer for highlighting the formatting inconsistencies in Tables 3, 4, 5, and 6. We acknowledge the issue regarding inconsistent sizes and font styles, and we will carefully address these presentation details in the refined version to ensure clarity and visual consistency throughout the paper. We appreciate the reviewer’s feedback, which will help improve the overall quality and professionalism of our refined version.

[1] Li, Chengzu et al. “Imagine while Reasoning in Space: Multimodal Visualization-of-Thought.” arXiv preprint, 2501.07542.

We thank the reviewer for the constructive feedback.

Q1: Impact of top-k filtering or nucleus sampling.

A:

• By default, we adopt nucleus sampling with p = 0.7 in our experiments. To explore the impact of the sampling parameter, we conducted experiments under different p values. As shown below, model performance is somewhat sensitive to the value of p. In particular, performance degrades when p is too small (e.g., p = 0.4), likely due to overly restrictive sampling that limits output diversity. | Nucleus sampling | MMIE | InterleavedBench | |------------------|------|-------------------| | p=0.7 | 59.5 | 3.13 | | p=0.9 | 59.1 | 3.04 | | p=0.4 | 58.4 | 2.81 |

• Experiments with top-k filtering are currently in progress. We will update the results as soon as they become available and would be happy to share them with the reviewer.

Q2: Clarity of theoretical analysis.

A: We appreciate the reviewer’s feedback regarding the clarity of our theoretical analysis. Below, we provide a simplified explanation of our key ideas:

• We propose a unified policy optimization framework (Eq. 3 and 4) that treats both text and image actions within the same policy space. This allows the model to learn a coherent generation trajectory across modalities. By jointly optimizing actions from different modalities, we ensure global consistency and enable smooth gradient flow through the interleaved sequence, which is essential for learning meaningful cross-modal dependencies.

• In previous methods (e.g., Deepseek-R1), defining accurate step-level rewards is difficult, often leading to ambiguous credit assignment and unstable training. In contrast, our interleaved multimodal setup naturally segments the generation process through explicit modality switches (e.g., text → image → text), which serve as clear and semantically meaningful boundaries. These transitions allow us to assign rewards to specific modality segments, improving the interpretability of the reward function, reducing gradient variance, and stabilizing learning.

We hope this explanation helps clarify the motivation and formulation of our method.

Please feel free to let us know if you have any further questions. We would be happy to address them.

Dear Reviewer 3D4S

Thanks again for your valuable comments and suggestions. As the discussion phase is nearing its end, we wondered if the reviewer might still have any concerns that we could address.

We believe our responses addressed all the thoughtful questions raised by the reviewer. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you！

Best wishes,

Authors

Weakness1: The experiment is only conducted on Vila-u, which is less powerful than other baselines as shown in Table 5 for multimodal understanding or generation, the generalization of the proposed method on the more powerful unified VLM is not demonstrated or proved.

A: To address this concern, we have conducted extensive experiments on Janus-Pro [1], a more powerful and unified VLM compared to Vila-u, with stronger performance on both multimodal understanding and generation tasks. As shown in the tables below, our proposed method consistently improves performance across various evaluation metrics on Janus-Pro, demonstrating its effectiveness and generalizability beyond a single model architecture.

Weakness2 & Q1: The compared baselines in main results (Table 1 and Table 2) are somehow outdated, which is lack of more powerful baselines like EMU3 to ensure a more comprehensive evaluation.

A: We thank the reviewer for the suggestion. However, not all unified models currently support interleaved generation, which is the core focus of our method. For completeness, we faithfully report the results of baseline models in Table 1 and Table 2 based on their original papers to ensure fairness and reproducibility.

Furthermore, we have additionally included the performance of EMU3 on standard benchmarks where applicable, to offer a more comprehensive view of model capabilities. Our method still demonstrates competitive or superior performance, further confirming its effectiveness.

Weakness3 & Q2: There is limited insight the community can gain after reading this paper. It is expected that after GRPO with combined rewards, the model can achieve better performance on interleaved generation for text and images, adding a baseline of SFT only post-train and comparing it with RL post-train may be more insightful.

A: We thank the reviewer for the valuable suggestion. In the main paper, we have included a comparison in Table 3 where the model is trained only with interleaved data (only w/ warm-up stage) using supervised fine-tuning. The results (53.31 on MMIE and 1.97 on InterleavedBench) show that SFT alone yields suboptimal performance on interleaved generation tasks, indicating that exposure to interleaved data is helpful but not sufficient for optimal performance.

Furthermore, to ensure a fair comparison, we extend the experiment by increasing the SFT data to match the scale used in RL post-training. As shown in the table below (Row 2), even under this setting, RL post-training with our combined reward design consistently outperforms SFT, which only gains minimal benefits. The results suggest that reinforcement learning brings non-trivial gains beyond what SFT can achieve, especially in aligning fine-grained generation behavior across modalities. We will highlight these insights more clearly in the revision to better convey the added value of our approach.

[1] Xiaokang Chen, et al. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint, 2025.

We appreciate the reviewer's time for reviewing and thanks again for the valuable comments and the positive score!

Weakness1: The proposed framework lacks novelty. The GRPO algorithm remains unchanged, still producing token-by-token outputs without any modifications. As a result, this framework is only applicable to autoregressive (AR)-based models such as Janus-Pro. It cannot be applied to other mainstream unified models such as Transfusion, MetaQuery, or BAGEL.

A:

1. We respectfully disagree with the reviewer’s comment that our GRPO framework is unchanged and lacks novelty. In this work, we make several substantial extensions beyond the original GRPO algorithm.

• We extend GRPO to the multimodal setting by introducing a unified policy optimization framework, which allows actions from the visual modality to be seamlessly integrated into the overall generation trajectory and jointly optimized alongside textual actions. This enables global optimization over the interleaved sequence, ensuring semantic consistency and facilitating effective gradient propagation across modalities.

• We further design a novel hybrid process-level reward that allows for finer-grained credit assignment and better alignment of intermediate generation steps. These innovations are non-trivial and crucial for effectively adapting GRPO to large-scale multimodal generation tasks.

2. Furthermore, our approach is not inherently tied to autoregressive models. We simply adopt VILA-U as our base model, which operates in an autoregressive fashion. In fact, GRPO strategies have also been successfully applied in diffusion- and flow-matching-based frameworks, such as Flow-GRPO [1].

• Mathematically, regardless of whether generation is performed via AR decoding or diffusion sampling, both paradigms can be abstracted as sampling from a parameterized policy $\pi_\theta(y \mid x)$, where $y$ denotes the generated output (either autoregressive sequence or diffusion sampling trajectory) and $x$ is the input context. Both approaches optimize $\pi_\theta(y \mid x)$ through gradient-based updates guided by relative preferences over grouped samples.

• Therefore, our framework—particularly the design of process-level hybrid rewards—is inherently model-agnostic, and can be seamlessly integrated into a wide range of generation paradigms beyond the AR setting. We will further investigate its applicability to diffusion-based or flow-matching-based generators in future work.

We hope our response helps clarify the novelty and contributions of our work, as well as eliminate potential misunderstandings—particularly regarding the misconception that our framework is limited to autoregressive models.

Weakness2: The use of multiple rewards is rather straightforward and does not constitute a significant contribution.

A:

1. First, we would like to clarify that developing new reward models is not the primary motivation of this work. Our core goal lies in building a unified framework under the GRPO paradigm that enables advanced interleaved multimodal generation. Therefore, we deliberately adopt multiple straightforward yet effective reward models as functional tools to guide policy learning. The innovation of our approach lies in leveraging multimodal hybrid reward signals that provide comprehensive supervision for interleaved multimodal generation.

2. Moreover, for credit assignment, we do propose a novel process-level reward specifically tailored for interleaved multimodal generation.

• Previous works, such as Deepseek-R1, attempted to use process-level rewards, but struggled with the challenge of defining “correct steps” in complex tasks. This ambiguity led to reward diffusion and noisy gradient updates, hindering stable learning.

• In contrast, interleaved multimodal generation offers natural, semantically grounded step boundaries via modality transitions (e.g., text to image), which act as anchors for defining local rewards. This enables clearer credit assignment along the trajectory and improves sample efficiency during optimization. Experimental results in Table 4 (+1.67 on MMIE and +0.34 on InterleavedBench) underscore the significance and contribution of the proposed process-level reward.

Therefore, we believe the choice of reward models does not undermine the novelty or contributions of our work; on the other hand, our innovation lies in how rewards are assigned—particularly through the proposed process-level design tailored for interleaved generation.

Weakness3 & Q3: The interleaved generation primarily focuses on coherence and creativity, but does not demonstrate capabilities such as interleaved reasoning or other complex interleaved scenarios, which limits its applicability.

A: We thank the reviewer for raising this point.

1. First, we would like to clarify that the benchmarks selected in our main experiments already include scenarios involving interleaved reasoning and other complex interleaved tasks. For example, we have reported the ``Multi-step reasoning" metric on MMIE benchmark in Table 1, and our results (53.28%) outperforms current advanced models.

2. To further strengthen this evaluation, we conduct additional experiments following the MVoT protocol [2], where a suite of maze-specific tasks is used to evaluate the model’s visual reasoning capabilities across multimodal contexts. As shown in the table below, our method achieves substantial performance gains on these benchmarks, further validating its ability to handle complex interleaved reasoning beyond simple coherence and creativity. |Method|MAZE|MINIBEHAVIOR|FROZEN-LAKE| |:----|:----|:----|:----| |MVoT|0.9295|0.9514|0.856| |Vila-U|0.7046|0.6527|0.5839| |Ours|0.9593|0.9706|0.8742|

Q1: Regarding Table 3, I would like to see the performance of the model when applying GRPO directly without the warm-up stage. While the model can generate interleaved outputs even without warm-up (albeit of poor quality), it would be valuable to understand how much the warm-up stage contributes to the final performance by comparing the direct GRPO approach against the full two-stage pipeline.

A: We appreciate the reviewer’s insightful suggestion. As requested, we conducted an additional experiment where we directly train the unified model using GRPO without the warm-up stage. The results (Row 2) show that generation quality after GRPO is significantly lower compared to the full two-stage pipeline. We attribute this to the model’s poor initial ability to generate interleaved outputs, which limits its capacity to produce high-reward samples during the GRPO stage. These findings underscore the importance of the warm-up stage in providing a reasonable initialization for modality transitions and content generation.

To further investigate this phenomenon, we explored an alternative strategy under the pure GRPO setting by explicitly encouraging the model to sample modality-switching actions through modifications to the sampling policy and reward formulation. Specifically, during the initial training phase, we manually sampled an interleaved content trajectory by adjusting the sampling probability of modality-switching tokens. Additionally, we introduced a multimodal content reward to enhance its relative advantage within the group and incentivize the generation of interleaved content.

However, performance (Row 3) still fell short compared to the warm-up+GRPO setup, indicating that starting from a cold initialization hinders the model’s ability to discover meaningful interleaved structures solely through reinforcement signals. These results support our two-stage training design and highlight the critical role of the warm-up stage in stabilizing training and guiding the model toward effective interleaved generation.

Q2: The ablation study on group size in GRPO presented in Table 6 appears limited in scope. Given that recent work such as DeepSeek-R1 has explored larger group sizes (8, 12, 16), it would be beneficial to extend this analysis to investigate whether larger group sizes could further improve performance. This expanded ablation would provide better insights into the scalability of the approach and help determine the optimal group size configuration for multimodal interleaved generation tasks.

A: We appreciate the reviewer’s suggestion and agree that exploring larger group sizes can offer valuable insights into the scalability of GRPO. However, due to the heavy memory usage of unified multimodal models, especially during group-wise sampling and interleaved generation, we were only able to extend our ablation study up to group size = 6 and 8 within our current hardware constraints.

As shown in the table below, increasing the group size to 6 yields noticeable improvements, indicating the potential benefit of larger groups. However, further increasing the group size to 8 brings only marginal gains. We acknowledge the value of further exploration at even larger group sizes (e.g., 12, 16), and we plan to include such extended experiments in future work when more computational resources become available.

[1] Liu, Jie, et al. "Flow-grpo: Training flow matching models via online rl." arXiv preprint, 2505.05470.

[2] Li, Chengzu et al. “Imagine while Reasoning in Space: Multimodal Visualization-of-Thought.” arXiv preprint, 2501.07542.

Dear Reviewer Ksvg,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our detailed explanations and additional results solve the concerns. We have addressed all the thoughtful questions raised by the reviewer (e.g., applying GRPO directly without the warm-up stage) and we hope that our work’s contribution and impact are better highlighted with our responses. As the discussion phase is nearing its end, it would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors

Reviewer Ksvg

Thanks again for the valuable comments and suggestions. As the discussion phase is nearing its end, we wondered if the reviewer might still have any concerns that we could address. We believe our responses addressed all the thoughtful questions raised by the reviewer (eg, novelty, reward, interleaved reasoning and ablation study).

It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you！

Best wishes,

Authors

Weakness1: The computation of the process-level reward (L200) is unclear to me. Is it computed using modality-specific segment as target output, and follows the same reward computation strategy as in Eq. 7?

A: Thank you for the question. In our approach, we treat modality-specific segments as the target outputs. However, the reward computation for each such step $Y^k$ slightly differs from Eq. 7. We compute a modality- and format-specific local reward $r^k$ as follows:

Specifically, the local reward is only computed based on the specific modality segment, and is then uniformly assigned to all tokens within that segment.

For credit assignment across steps within the trajectory, we follow a group-wise outcome comparison strategy similar to outcome-level supervision in Eq. 5:

We normalize the per-step scores within the group, and compute the relative advantage of each step. This strategy enables a more fine-grained and diverse credit assignment across modality-specific steps. We will clarify this implementation detail more explicitly in the refined version of the paper.

Weakness2: In Table 5, the proposed method underperforms compared to VILA-U on MMVet and POPE.

A: We thank the reviewer for pointing this out. The slightly lower performance of our method on MMVet and POPE compared to VILA-U is within a reasonable margin of experimental variance (approximately 2% and 1%, respectively), and we believe it does not indicate a fundamental limitation of our approach.

To further address this concern, we conducted additional experiments using Janus-Pro, a stronger unified model. The results demonstrate consistent improvements across all benchmarks, including MMVet and POPE, validating that our GRPO framework remains effective and generalizable on these benchmarks as well.

Q1: I wonder if the authors have tried the version without the warmup stage, how does the model perform?

A: We appreciate the reviewer’s insightful suggestion. As requested, we conducted an additional experiment where we directly train the unified model using GRPO without the warm-up stage. The results (Row 2) show that generation quality after GRPO is significantly lower compared to the full two-stage pipeline. We attribute this to the model’s poor initial ability to generate interleaved outputs, which limits its capacity to produce high-reward samples during the GRPO stage. These findings underscore the importance of the warm-up stage in providing a reasonable initialization for modality transitions and content generation.

To further investigate this phenomenon, we explored an alternative strategy under the pure GRPO setting by explicitly encouraging the model to sample modality-switching actions through modifications to the sampling policy and reward formulation. Specifically, during the initial training phase, we manually sampled an interleaved content trajectory by adjusting the sampling probability of modality-switching tokens. Additionally, we introduced a multimodal content reward to enhance its relative advantage within the group and incentivize the generation of interleaved content.

However, performance (Row 3) still fell short compared to the warm-up+GRPO setup, indicating that starting from a cold initialization hinders the model’s ability to discover meaningful interleaved structures solely through reinforcement signals. These results support our two-stage training design and highlight the critical role of the warm-up stage in stabilizing training and guiding the model toward effective interleaved generation.

Dear Reviewer 7qgK,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our detailed explanations and additional results solve the concerns. We have addressed all the thoughtful questions raised by the reviewer (e.g., directly train using GRPO without the warm-up stage) and we hope that our work’s contribution and impact are better highlighted with our responses. As the discussion phase is nearing its end, it would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors