Generative Visual Instruction Tuning

We thank the reviewer for the feedback and comments. Below are our responses.

Limited Innovation

We emphasize the following to be our key contributions:

• Model Representation : Unlike GILL, which maps LLM embeddings to text embeddings of the Stable Diffusion model, our method directly injects LLM representations into the visual latent space of the diffusion process. This distinction allows GenLLaVA to enable image editing capabilities directly influenced by visual context rather than textual abstraction.

• Model Architecture : Our approach is distinct from Unified-IO 2 (Encoder-Decoder) as it adopts a decoder-only architecture, reuses existing models, and does not require training from scratch, thereby significantly reducing computational costs.

• Dataset Contribution : We compiled a novel fine-tuning dataset that integrates multimodal instruction tuning data for image understanding, generation, and editing. Unlike existing works, this dataset enables researchers to bypass the need for collecting and combining individual datasets manually.

• Q-Former Utilization : While superficially similar to prior work, the Q-former enables better integration with latent diffusion and improves generative performance, as reflected in our ablations in Section 4.6.

Baseline Comparison

We appreciate the reviewer's suggestions regarding the choice of baselines and have expanded the evaluation to include more models, such as Emu, SEED-X, Show-O, and DreamLLM in the following table:

Table B: Performance comparison of GenLLaVA against state-of-the-art models on multimodal understanding and generation tasks. We will fill in the missing values in the table for the revised manuscript.

Some models mentioned by the reviewer, such as Mini-Gemini, are not true generative models, as they rely on creating detailed prompts for a diffuser (akin to tool use).

Unfounded Claims

We will qualify our claims to indicate how we make progress on this problem as we are being made aware of other efforts and we contextualize our work further (see Table 1 and Section 4.7 of the manuscript for more details).

Thanks for the authors' response. However, I still concern about the novelty of this paper considering a large number of advanced models introduced this year that can achieve unified image understanding, generation, and editing. I decided to keep my rating.

We thank the reviewer for the feedback. Below are our detailed responses.

Incremental Nature of the Work We emphasize the following key distinctions from prior work, specifically SEED-X and Unified-IO 2:

• Model Training : Unlike Unified-IO 2, which is trained from scratch with an encoder-decoder architecture, our model adopts a decoder-only architecture and reuses existing models, reducing computational costs while maintaining good performance.

• Dataset Contribution : We compiled a novel fine-tuning dataset for multimodal tasks (understanding, generation, and editing). Unlike SEED-X, which only releases an image editing dataset, our dataset integrates these modalities seamlessly.

• Omnimodality : SEED-X relies on task-specific checkpoints (SEED-X-I, SEED-X-Gen, SEED-X-Edit), forcing users to switch models for different tasks. While SEED-X includes a general-purpose checkpoint, it significantly underperforms across tasks. In contrast, GenLLaVA balances all modalities, achieving good performance without task-specific checkpoints.

• Generation Limitations : SEED-X relies heavily on CLIP features for generation, which inherently are not fine-grained enough for high-quality Generation. GenLLaVA overcomes this by integrating features generated by the LLM, enabling finer control and better Generation capabilities.

Domain-Expert Comparisons: Including more domain-expert baselines would provide a comprehensive perspective. We will include additional results for domain-specific models (e.g., MGIE for image editing) in Table C and the supplementary material. These results contextualize the performance of GenLLaVA as a unified model able to balance these tasks whitin the same model.

Table C: Performance comparison of domain-specific experts and GenLLaVA across various tasks. Img. Gen Expert is SD1.4, Img. Editing Expert is MGIE, and Img. Und. Expert refers to a LLaVA model trained on the image understanding data we collected.

User Intent Understanding: The use of task tokens ([T2I], [I2T]) is a compromise for balancing performance across tasks. Inferring user intent remains an open challenge, as evidenced by the limitations of purely omnimodal systems like Chameleon, which sometimes fail to distinguish between tasks effectively. Other strategies, such as modality-specific checkpoints (e.g., Emu models, SEE-X), bypass intent inference entirely but compromise generality.

Our approach aligns with other state-of-the-art models, such as Unified-IO 2, Show-O, and DreamLLM, which also utilize task-specific tokens. This strategy explicitly conditions the model during training and inference, reducing task ambiguity, providing the user with a way to seamlessly change tasks without having to re-load a new chekpoint.

We thank the reviewer for their comments. Below are our responses:

Limited Originality in Methodology

• Model Design and Contributions :

  • Our model is a decoder-only architecture that reuses existing pre-trained components (Mistral LLM, SigLIP vision encoder, and Stable Diffusion), which reduces computational requirements compared to training from scratch (e.g., Unified-IO 2).

  • Unlike prior works such as GILL, which guide the diffusion model solely via text prompts, GenLLaVA directly feeds features from the LLM into the U-Net visual encoder of the diffusion model. This enables direct visual guidance during diffusion, facilitating tasks like image editing. In contrast, text-only guidance (e.g., MiniGemini) necessitates intermediate text conversion, limiting editing capabilities.

  • Models like SEED-X and the Emu family require task-specific checkpoints (SEED-X-I, SEED-X-gen, SEED-X-edit) for understanding, generation, and editing, reducing generalization. GenLLaVA maintains a unified model with competitive performance across tasks, providing the user with a way to seamlessly change tasks without having to re-load new checkpoints.

• Dataset Contributions :

  • We curated a multimodal instruction-following dataset that extends LLaVA-Finetune with new generative and editing tasks, offering a comprehensive, reusable resource for the community.

Preference for Single-Stage Training Our decision for a single-stage training pipeline stemmed from the following observations:

• Empirical Performance :

  • Multi-stage pipelines, including recipes that separate multimodal alignment, generative capability addition, and instruction tuning, resulted in catastrophic forgetting. For example, when multimodal alignment was conducted first, subsequent stages significantly degraded the model’s understanding performance (see Table 1 in the manuscript for results).

  • Single-stage training, as shown in prior works like PrismaticVLMs, simplifies the process while maintaining competitive results for both understanding and generative tasks. We remark that, we only cite PrismaticVLMs, as the first to do so but only for image understanding tasks, while we extend this to image generation and editing tasks.

• Practical Advantages :

  • Single-stage training reduces computational costs by approximately 20% and enables us to leverage existing datasets like LLaVA-Pretrain without the need for additional alignment data collection.

We have added comparisons with different training strategies to Table D for further clarity.

Table D: Different recipes for training. We assume that the regular alignment stage is done first using LLaVA-Pretrain-558K data except for GenLLaVA since it is single-stage recipe.

Performance Gaps with State-of-the-Art Models

With access to larger-scale understanding and generative datasets, we hypothesize that GenLLaVA’s performance could improve significantly. However, quantifying the exact data requirements to achieve the same results would be nothing more than speculation. See also limitations sections where we compare to other state-of-the-art models

Open-Sourcing Resources

We will release all resources, including datasets, code, and model checkpoints, upon acceptance. The data reuses the LLaVA-Pretrain and LLaVA-Finetune formats, with additional fields for generation tasks (e.g., output_image).

We thank the reviewer for the feedback and comments. Below are our responses.

Conciseness of the Methodology Section

In the revised manuscript, we will:

• Elaborate on the loss functions used for both visual understanding and generation tasks.
• Provide details on how classifier-free guidance (CFG) is used in our model to combine outputs from the large language model and the Diffuser model.
• Clearly outline the configuration of attention masks and the specific inputs and target outputs during the image generation and editing phases.

Limited Originality in Methodology

We emphasize the following to be our key contributions:

• Model Architecture : Our approach is distinct from Unified-IO 2 (Encoder-Decoder) as it adopts a decoder-only architecture, reuses existing models, and does not require training from scratch, thereby significantly reducing computational costs.
• Dataset Contribution : We compiled a novel fine-tuning dataset that integrates multimodal instruction tuning data for image understanding, generation, and editing. Unlike existing works, this dataset enables researchers to bypass the need for collecting and combining individual datasets manually.
• Diffuser Model Integration : GILL uses a less capable LLM (OPT-6.7B) and noisier data (CC3M). Unlike GILL, which relies on text prompts to guide the diffusion model, GenLLaVA directly injects LLM features into the visual encoder (U-Net), enabling visual guidance and editing capabilities. Text-only guidance limits editing, as visual input must first be converted into text-embedings.

Experimental Support for Claims

Some trade-offs in understanding performance are observed. To address this:

• We will clarify that our claim is specific to maintaining a balance across modalities, not absolute superiority in all metrics.
• The observed performance gap (e.g., a 2% difference on MMB) is within an acceptable margin, demonstrating that the trade-offs are minor. We will qualify our claims to make this observation.

Single-Stage Pipeline Justification

Our choice of a single-stage pipeline was motivated by empirical observations and challenges with multi-stage pipelines, we only cite PrismaticVLMs, as the first to do so but only for image understanding tasks, while we extend this to image generation and editing tasks.

• Multi-stage recipes (e.g., separating multimodal alignment, generative capability addition, and instruction tuning) often resulted in catastrophic forgetting, severely degrading performance on tasks. For example, two three-stage pipelines yielded poor results, see Table A.

Inadequacy of Ablation Studies

We will strengthen our ablation study section by:

• Adding a comparison of task-specific performance with and without additional understanding data. See Table A for specific results.
• Highlighting the role of task tokens in mitigating inter-task conflicts. Task tokens explicitly define the required task during inference, reducing ambiguity and enhancing task-specific performance. This is consistent with findings from similar models like Unified-IO 2 and DreamLLM, Show-O, AnyGPT all of which use analogus strategies.

Role of Task Tokens

Task tokens serve as a simple yet effective strategy to condition the model explicitly for each task, bypassing the need for the model to infer user intent. This prevents errors in multi-modal understanding tasks as greatly present in Chameleon (no task tokens). Other strategies, such as releasing task-specific checkpoints (e.g., Emu models that release generation and understanding checkpoint separately), achieve similar goals but sacrifice the model's generality. We will expand the discussion of this mechanism in the manuscript.

Specific Performance Comparison for Image Understanding

Section 4.7 already includes comparisons of our model against state-of-the-art methods for multimodal understanding tasks. However, we will further elaborate on how the added data enhances image understanding. Specific results are detailed in Table A.

Table A: Different recipes for training. We assume that the regular alignment stage is done first using LLaVA-Pretrain-558K data except for GenLLaVA since it is single-stage recipe.