OmniGen-AR: AutoRegressive Any-to-Image Generation

Q1: Comparison with existing models
A1: Thank you for pointing this out. We appreciate the reviewer bringing EditAR to our attention and agree that it is relevant work that should be cited and discussed. We will include a citation and clarify the distinctions in the revised version of the paper:

While EditAR and our method both employ autoregressive transformers and support multiple conditional image generation tasks, they are fundamentally different in two aspects: 1) EditAR is specifically designed for image editing and low-level control tasks (e.g., depth-to-image, edge-to-image, segmentation-to-image), while OmniGen-AR is a unified Any-to-Image framework that handles a broader range of input modalities. 2) EditAR aims to improve text-image alignment by introducing distillation loss, while our model hopes to prevent information leakage through DCA, a novel training-time attention mechanism that disentangles condition and content attention paths.

Q2: More results on text-to-image, frame prediction, and depth-to-image
A2: Thanks! We will add more quantitative results in the revised paper:

Q3: Empirical comparison between DCA and classifier-free guidance
A3: Thanks for your great suggestion! The proposed DCA is fundamentally different from CFG, as it does not directly drop the conditions. Instead, it calibrates the attention between condition and content tokens to allow the model better disentangle and leverage conditional information. Below we show empirical comparison between them. As can be seen, compared to CFG, DCA leads to significantly better results on tasks like frame prediction, editing, and controllable generation.

Q4: Analysis of the discrepancy with ControlAR
A4: We acknowledge that ControlAR achieves higher mIoU on both mask-to-image and depth-to-image tasks. However, it’s important to highlight that ControlAR is specifically designed for controllable generation and relies on an additional control encoder (initialized from DINOv2-S), which is pretrained for dense visual understanding. This encoder introduces strong spatial priors that benefit structured control tasks but are task-specific and less generalizable to other modalities.

In contrast, OmniGen-AR maintains a unified architecture without any task-specific heads or encoders, supporting a wider variety of inputs, including text, segmentation, depth, canny edges, image editing, and video frame prediction, all using a single causal transformer. While we may not outperform ControlAR on a few specific spatial tasks, we offer greater flexibility, simplicity, and generality, which are essential for Any-to-Image generation.

Q5: Visualization of editing and spatial-conditioned datasets
A5: Thanks! We will add this to our revised paper.

Q6: Analysis of the effects of DCA probabilities
A6: We thank the reviewer for raising this point. While DCA helps prevent condition-content leakage during training, applying it too frequently disrupts the model’s ability to fully leverage causal dependencies, especially when condition tokens carry rich information (e.g., dense segmentation or image context). In our experiments, 10% DCA strikes a balance: it encourages robust conditioning without significantly limiting the model’s access to informative context. We will update the paper to include more detailed analysis.

Q1: The claim of "the first AR for Any-to-Image generation"
A1: Thank you for the valuable comment. We agree that there have been prior autoregressive models that support multiple input modalities, including text and image. However, our goal is to develop a unified autoregressive visual generation model that explicitly accommodates three types of conditions, text, spatial, and visual contex, in a single framework. While models like Unified-IO and Unified-IO 2 indeed support text and spatial inputs, they do not explicitly unify visual context as a generative condition. Lumina-mGPT is trained with diverse inputs, but to the best of our knowledge, it only reports performance on text-to-image generation, without demonstrating generalization to the broader range of input conditions explored in our work. We will revise our claim in the paper to better reflect this nuance.

Q2: The claim of "achieving state-of-the-art on benchmarks"
A2: Thank you for your comment. We will revise the paper to more accurately characterize our results in comparison to prior work, below we would like to provide some important context regarding the comparisons mentioned: 1) diffusion models like [5] typically rely on additional text encoders that are typically not counted in the model size. For example, [5] uses a strong 2B-parameter text encoder (Gemma-2B), which is non-negligible compared to the diffusion models themselves. In contrast, our model uses a single causal transformer without external text encoders, keeping the architecture clean and parameter-efficient. 2) AR model [4] and hybrid model [6] are trained on much more data than our model. Specifically, our model is trained on approximately 56M text-to-image pairs and 13M examples for image editing and control generation. In contrast, [4] reports using around 140M samples, while [6] uses roughly 2B examples. Despite this significant parameter or data gap, our model achieves competitive performance across a range of benchmarks, especially given our unified support for diverse conditional generation tasks.

Q3: Novelty of this work
Thank you for the thoughtful comments. We would like to clarify the novelty and motivation of our work: we do not seek to alter the autoregressive modeling manner, but to build a unified AR framework that accommodates diverse conditional inputs. To alleviate the information leakage issue from condition tokens to content tokens in causal modeling, we introduce Disentangled Causal Attention (DCA), a training-time regularization scheme that carefully preserves the causal nature of AR generation while enabling the model to learn condition-aware generation without overfitting or shortcutting. Though lightweight by design, DCA is proved to be effective in improving the instruction following capability of our model, as evidenced by our ablation studies.

Q4: The advantage of using AR for image generation
Thanks for pointing this out. Our choice of an autoregressive (AR) model is motivated by its ability to flexibly accommodate diverse input conditions within a single causal transformer, in addition to the advantage you noted. There is currently no clear consensus in the community regarding which family of generative models, AR, diffusion, or VAR, is fundamentally superior to visual generation. Each has its own trade-offs in terms of quality, extensibility, and inference speed. We believe AR models remain an important and under-explored direction, especially in the context of multi-conditional visual generation, and our work aims to push this frontier forward.

Q5: Qwen initialization
Thank you for the question. We fine-tune a pretrained Qwen2.5 model for image generation tasks. Specifically, we adopt Qwen2.5 both as the text tokenizer and as the initialization for the transformer decoder. During training, we compute the autoregressive loss on the entire sequence, including both the text tokens (input prompts) and the visual tokens (generated image latents).

• Performance with and without Qwen initialization: we conducted experiments using both Qwen-initialized and randomly initialized models and found that both setups can achieve similar performance on text-to-image (T2I) benchmarks after convergence (0.29 vs 0.31 on GenEval after 512 resolution pretraining). However, Qwen initialization leads to more stable training in the early stages, with lower loss.
• Natural language understanding performance after T2I fine-tuning: after fine-tuning Qwen2.5 on T2I data, we observe a significant drop in language understanding performance, with MMLU accuracy decreasing from 47.5 to 12.7. This degradation is expected as we do not include any text-only data during fine-tuning. Although we compute loss on text tokens, the learning signal is relatively simple and repetitive, and lacks the diversity and complexity of language modeling tasks. We believe that the text understanding capability can be better preserved by mixing in text-only data for joint training, and we leave a systematic study of its impact on both language and T2I performance for future work.

Q6: Inference speed
A6: Thank you for the question. We evaluate the inference speed of our model alongside other models of comparable scale. Since Janus only supports 384×384 resolution, we only report its speed based on 24×24 tokens. As shown below, benefiting from the usage of KV cache, the inference speed of our model clearly outperforms Show-o. While our generation speed is still slower than SANA, deploying with vLLM could significantly narrow the gap.

Dear reviewer, thank you again for your insightful comments on our paper, and we genuinely hope that our response could address your concerns. As the discussion is about to end, we are sincerely looking forward to your feedback. Please feel free to contact us if you have any further inquiries.

Hi Reviewer EJMS, since the reviewer-authors discussion time window will be closed soon, can you help read the rebuttal as soon as possible so that we can address any remaining concerns you may have. Grateful for your effort!

Q1: Weaker results on some metrics and further analysis
A1: Thanks for pointing this out! We acknowledge that our model shows weaker performance on segmentation-to-image in terms of mIoU compared to ControlAR and OmniGen, and will add more analysis in the revised paper.

• For ControlAR, it leverages a dedicated control encoder initialized from DINOv2-S, which provides strong representations tailored for visual understanding. In contrast, our model uses a unified causal transformer without condition-specific encoders, aiming for architectural simplicity and generality across a wide range of tasks—not only spatial control but also text, video, and editing-based generation. Moreover, ControlAR is limited to controllable generation tasks, while our framework supports a broader set of inputs, including text-to-image, image editing, depth, segmentation, and even text-to-video.
• For OmniGen, the performance advantage stems in large part from the scale and diversity of its training data. OmniGen is trained on a massive self-constructed dataset, covering a wide spectrum of tasks including text-to-image (~80M), image editing, human motion, virtual try-on, style transfer, subject-driven generation (e.g., GRIT-Entity, Web Images), and structured control generation (e.g., LAION with depth, segmentation, pose, and canny annotations, RefCOCO, ADE20k, and ReasonSeg). In comparison, our model is trained on a more modest dataset: ~56M T2I samples and ~13M for editing and control generation, using only publicly available sources. Despite these differences, our model still achieves strong and competitive results across most benchmarks, while maintaining a clean, unified autoregressive architecture that handles diverse input modalities without task-specific heads or encoders. We believe this highlights the scalability and generality of our approach.

Q2: The analysis in Section 4.4 is minimal, leaving unclear how to address observed editing failures
A2: Thank you for pointing this out. We will expand Section 4.4 in the revised version to include a more detailed analysis of failure cases: As shown in Figure 8, failure cases can be broadly categorized into two types: 1) Instruction-following capability. For instance, in the first row of Figure 8, the instruction is “Remove the bag on the bench next to the person sitting at the bus stop”, but the model removes the person instead. This indicates a failure in grounding fine-grained spatial and referential cues from language into visual modifications. 2) Low-quality generations under sparse control signals. Examples in the second row (depth-to-image and segmentation-to-image) show blurry or structurally inconsistent results, which likely stem from noisy supervision and sparse training coverage for these conditions. These failure modes suggest two potential directions for future work: 1) Scaling up the model and training data to build a stronger base model with improved generalization and instruction-following ability across diverse visual tasks. 2) Leveraging chain-of-thought (CoT) to improve the model’s reasoning ability on complex prompts.

Q3: The rationale of DCA mechanism
A3: Thank you for your comment. Disentangled Causal Attention (DCA) is designed to address a concrete and critical issue in conditional autoregressive generation: the risk of information leakage from condition tokens to content tokens during training. In standard causal attention, the model may inadvertently learn to exploit trivial correlations or positional cues between condition and target tokens, rather than learning meaningful conditional generation.

DCA introduces a principled modification to the attention mask, splitting it into condition and content causal regions. This separation enforces a clearer dependency structure, encouraging the model to learn robust alignment between inputs and outputs without relying on token overlap or order-based shortcuts. While DCA is simple in form, it is grounded in a clear rationale: to preserve the autoregressive training signal while regularizing the attention pathway in a way that reflects the asymmetric role of condition versus content tokens. Its effectiveness is consistently validated across multiple conditional tasks, as shown in our ablation results, and it plays a key role in enabling our model to generalize across diverse inputs.

Q4: Discussion on training cost, sampling latency, or practical deployment of a large AR model
A4: Thanks for your valuable suggestions! We will include the discussion on training cost, sampling latency, and practical deployment in our revised paper:

• Training cost: We train our model on 64 A100 GPUs. The 512 pretraining of 0.5B/1.5B models took 6/10 days separately, while the 1024 sft stage took 4/6 days.
• Sampling latency and practical deployment: below we show that the AR visual generative model is compatible with the vLLM framework and deploying with it could significantly improve the inference speed.

Dear reviewer, thank you again for your insightful comments on our paper, and we genuinely hope that our response could address your concerns. As the discussion is about to end, we are sincerely looking forward to your feedback. Please feel free to contact us if you have any further inquiries.

Q1: Total amount of training data and a brief comparison with other methods
A1: Thanks for your suggestion, below we compare the training data with several baselines:

Q2: Ablation studies on using loss constraints to prevent information leakage
A2: Great suggestion! We conducted experiments to apply loss reweighting between condition and content tokens—specifically, assigning a lower weight (0.1) to condition tokens and a standard weight (1.0) to content tokens during training. The results below show that the proposed DCA outperforms loss reweighting on all benchmarks by clear margins.

Q3: Quantitative experimental comparisons between the 1.5B model and the 0.5B model
A3: Below we report more quantitative experimental comparisons between 0.5B and 1.5B models, 1.5B model consistently achieve better results than 0.5B model.