Effective and Efficient Masked Image Generation Models

We thank reviewer 5pkV for the interest and acknowledgement of our contributions and the valuable comments. We respond below to your questions and concerns.

Minor Limit Generalizability of improvements beyond tested configurations unclear

Empirical analysis is a well-established approach in developing complex models. Prior works [a, b] similarly employed ablation studies and targeted experiments to optimize their designs. Through systematic testing of components and hyperparameters, they demonstrated the value of rigorous empirical analysis, even when optimal settings may be context-dependent. Building on this precedent, our study validates specific configurations and provides a framework for systematically exploring and understanding design choices in masked generative models. We believe this methodological contribution offers significant value to the research community.

Besides, eMIGM performs effectively with both VAE (ImageNet 256x256) and DC-AE (ImageNet 512x512) tokenizers despite their distinct latent spaces. This adaptability highlights the transferability of our design principles across different data representations.

[a]“Elucidating the Design Space of Diffusion-Based Generative Models.” [b]“Analyzing and Improving the Training Dynamics of Diffusion Models.”

Exp1: More evaluation metrics

We've now evaluated our models using additional metrics (sFID, IS, Precision, and Recall). These comprehensive results are presented in our response to reviewer DbgT's Weakness2 and will be included in the revised paper.

Sugg1: novelty and analysis of the choices of training

We sincerely appreciate your recognition of our unified framework, well-structured experimental design and results.

Regarding your concerns about novelty and analysis, we would like to clarify two key points:

1. Our hyperparameter tuning follows an empirical approach, aligning with established practices in VAR and MAR. This practical methodology remains valuable as it yields effective, demonstrable results.

2. Our modifications stem from careful analysis of MDM's behavior. For instance, the time interval strategy was motivated by our observation that MDM's generation process is irreversible, making early-stage guidance less effective - a finding we quantitatively validated in Appendix C. Similarly, replacing fake class tokens with mask tokens in CFG was informed by the fact that MDM has seen more mask tokens during training, making them a more natural choice than the fake class tokens used in diffusion models. We appreciate your suggestion and will expand our analysis of these design choices in the revision.

Our key contribution is a unified framework integrating masked generative transformers and masked diffusion models, advancing masked generative modeling through systematic experimentation and analysis.

Q1: Further training of Figure 2(e)

To reponse this question, we trained standard CFG for 800 epochs. Below we compare its FID scores with mask CFG:

The results show both approaches achieve equivalent final performance, with CFG with Mask converging more rapidly as shown in Fig.2(e) of our paper.

Q2: eMIGM vs Consistency Models

Consistency models and eMIGM employ distinct approaches to generation. Consistency models establish direct mappings from noise to data via consistency constraints, enabling single-step sampling. In contrast, eMIGM utilizes masked prediction, beginning with fully masked images and iteratively revealing content through progressive unmasking. This architectural distinction enables eMIGM to optimally balance generation quality and computational efficiency by modulating function evaluations.

Q3: choice of mask schedule

We explored three mask schedules (linear, cosine, and exponential) with constraints $\gamma_0 \approx 0$ and $\gamma_1 \approx 1$. While linear schedules are common in MDM text generation, we found concave functions like cosine performed better for images due to their information redundancy - higher mask ratios during training provide stronger learning signals. With $w(t)=1$, the exponential schedule slightly outperformed cosine, becoming our default.

Based on your suggestion, we developed a log-exp schedule ($\gamma_t=\frac{\log\left(1 + (e^5 - 1) \cdot t\right)}{5}$) that balances mask ratios by reducing both high and low masking extremes. Using the setup from Figure 2(b) with $w(t)=1$, we present results of FID below.

The log-exp schedule shows better convergence and performance, validating the benefits of exploring new masking approaches. We appreciate this insightful suggestion and will incorporate these findings in our revision.

We thank reviewer DbgT for the interest and acknowledgement of our contributions and the valuable comments. We respond below to your questions and concerns.

Claims 1: Define "comparable" more rigorously

To quantify our comparison: eMIGM achieves an FID of 1.57 on ImageNet 256x256 generation versus REPA's 1.40 - a 12% gap. We consider this competitive since eMIGM requires fewer function evaluations (128×1.4 NFEs vs. REPA's 250×2 NFEs). We will clarify this comparison more precisely in our revised paper.

Experiment1: Include addtional baselines (e.g. GANs)

Our Tables 2 and 3 already include comparisons with GAN baselines (BigGAN and StyleGAN-XL), which we will emphasize more clearly in our revision.

References Missing Meissonic

We appreciate the reference suggestion and will include it in our final paper.

Weakness1: Limited discussion of failure cases: What types of images does eMIGM struggle with?

While we have not observed distinct failure patterns, eMIGM does experience occasional generation failures, as do other methods like MAR and VAR. We will include representative failure cases in our revision.

Weakness2: Qualitative comparisons beyond FID

We provide qualitative visual comparisons with diffusion models and VAR at link and quantitative evaluations using sFID, IS, Precision, and Recall (see Table below), both of which will be included in our revision.

Q1: How does eMIGM perform on out-of-distribution datasets and different image distributions?

eMIGM demonstrates strong adaptability across data representations, performing effectively with both VAE (256x256) and DC-AE (512x512) tokenizers despite their different latent spaces, showing our efficiency improvements generalize across resolutions and distributions.

Regarding out-of-distribution, while important, our research focuses primarily on improving the efficiency and quality of the generative models themselves, following the established research direction of prior works like MAR and VAR. These foundational works concentrate on in-distribution generation, which remains our focus for high-quality image synthesis.

Q2: What limitations does eMIGM have compared to continuous diffusion models? When do diffusion models outperform eMIGM?

Compared to continuous diffusion models, eMIGM has limitations in zero-shot classification [a], ultimate generation quality with sufficient NFEs [b], and applications in video/audio synthesis. However, we believe masked generative models can narrow this gap as they evolve. We will discuss these limitations comprehensively in our revision.

[a]“Your Diffusion Model is Secretly a Zero-Shot Classifier.” [b]“Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps.”

Q3: Hybrid approach with diffusion models and modifications?

A hybrid approach combining eMIGM with diffusion models could enhance performance. Recent works [c,d] show that integrating autoregressive and diffusion models achieves superior generation capabilities compared to pure continuous diffusion models. Additionally, simple modifications could bridge remaining performance gaps, such as adopting rectified flow [e] for the diffusion head's training objective or following REPA [f] by aligning eMIGM's intermediate Transformer outputs with pretrained DINOv2 encoder features. We will incorporate this discussion in our revised paper.

[c]“Efficient Visual Generation with Hybrid Autoregressive Transformer” [d]“Diffusion Transformer Autoregressive Modeling for Speech Generation” [e]“Learning to Generate and Transfer Data with Rectified Flow” [f]“Training Diffusion Transformers Is Easier Than You Think”

Q4: How does eMIGM's robustness to adversarial perturbations compare to diffusion models?

We appreciate your insightful question regarding adversarial robustness. While we acknowledge the critical importance of this aspect in the broader context of generative modeling, our current research primarily focuses on enhancing the efficiency and quality of the generative models themselves. We acknowledge that our expertise in adversarial perturbations is limited. Currently, we have not performed a assessment of eMIGM's robustness against such perturbations. We will discuss this important aspect in our revised paper.

We thank reviewer 7cuF for the interest and acknowledgement of our contributions and the valuable comments. We respond below to your questions and concerns.

Claim1: using a weight schedule for guidance

Compared to existing work, our approach is motivated by MDMs' unique irreversible token generation constraint. We found that early strong guidance restricts generation diversity and increases FID scores. We will revise our paper to properly acknowledge prior work and clarify our specific contributions to guidance strategies for MDMs.

Claim2 and Weak1: unsupervised guidance method

We will clarify that we adapt the unsupervised CFG from text generation [4] to image generation. For clarity, we rename it to "CFG with Mask" to better reflect our focus on masked image generation. Besides, we will add a discussion of [5] in the final version.

Weak2: number of NFEs remains relatively high

Our work focuses on improving masked image modeling efficiency. For example, in 512x512 generation, eMIGM-L matches MAR's performance with fewer NFEs (64x1.25 vs 256x2). We believe future work applying distillation could further improve efficiency.

Weak3: Tables 2 and 3 are not entirely fair

While we previously cited results from the original EDM2 paper, we will update our comparison to include EDM2 with Guidance Interval [2].

Weak4: NFE comparison with EDM2

We conducted additional experiments comparing sampling speeds on a single A100 GPU (batch size 256). As shown below, eMIGM achieves faster sampling than EDM2 while maintaining competitive FID.

Sugg1: More discuss about weighting strategy and masking schedule

Our experiments showed that weighting functions significantly affect noise schedule performance. Using $w(t)=\frac{\gamma_t'}{\gamma_t}$ led to unstable training, especially with the exp schedule. Switching to $w(t)=1$ stabilized training across all schedules and improved performance, with the exp schedule yielding the best results. We adopted this combination as our default and will clarify this relationship.

Sugg2: About additional sampling cost

In our 512x512 image generation experiments with eMIGM-L (NFE=64x1.25), the diffusion model need 14 sampling steps using DPM-Solver, introducing approximately 14% additional sampling speed on a single A100 GPU (batch size 256). This is more efficient than MAR, which requires 100 diffusion steps. As shown in Response to W4, eMIGM achieves faster sampling (0.165s vs 0.552s per image) while maintaining competitive FID scores compared to EDM2. We will include a efficiency comparison table in our final version.

Sugg3: Fig.4 is misplaced

We will relocate Fig.4 to be positioned near Section 6.1.

Q1: Experimented with other weighting and masking functions

Regarding to the mask schedules, we explored three mask schedules (linear, cosine, and exponential) constrained by $\gamma_0 \approx 0$ and $\gamma_1 \approx 1$. In response to your question, we introduce a log-exp schedule that balances mask ratios by reducing extreme cases of both high and low masking in exp schedule. The schedule is defined as: $\gamma_t=\frac{\log\left(1 + (e^5 - 1) \cdot t\right)}{5}$. Following the experimental setup in Figure 2(b) with $w(t)=1$, we present comparative FID in the table below.

The log-exp schedule shows better convergence and performance, validating the benefits of exploring new masking schedule.

Regarding weighting functions, we explored two primary options: $w(t) = \frac{\gamma_t^\prime}{\gamma_t}$ and $w(t) = 1$. The log-normal weighting from EDM isn't directly transferable to our framework. Exploring alternative weighting functions remains promising for future work.

Q2: Performance with <16 sampling steps

We evaluated eMIGM's performance with fewer sampling steps on ImageNet 256×256, with results shown in the table below:

Q3: Could diffusion component benefit from cfg

Our diffusion component already implements classifier-free guidance. During sampling, our model processes both class and mask token inputs to generate $z_c$ and $z_u$ outputs. CFG guides generation toward class-conditioned results using the formula $\epsilon=\epsilon_\theta(x_t|t,z_u)+\omega\cdot(\epsilon_\theta(x_t|t,z_c)-\epsilon_\theta(x_t|t,z_u))$, where $\omega$ is the guidance scale. We will clarify this in our final paper.

Q4: model receive the class condition as input

The model still receives class conditions during training, but we randomly replace the true label with a mask token at a fixed probability. We will clarify this in our revised paper.

Claims: Some claims could be further strengthened by additional analyses.

We appreciate your feedback regarding our efficiency claims.

Our analysis of training efficiency examined the relationship between training FLOPs and FID scores. As shown in Fig.4(b), larger eMIGM models achieve better FID scores at equivalent FLOP budgets—for example, eMIGM-L outperforms eMIGM-B when both consume approximately 10^20 FLOPs, confirming superior training efficiency of larger models.

Regarding sampling efficiency, we evaluated the inference speed-FID trade-off (Fig.4(c)), with measurements conducted on an A100 GPU using batch size 256. Results demonstrate that larger eMIGM consistently deliver better FID scores than smaller models at comparable inference speeds.

We will follow your suggestion to enhance the clarity of these findings in our revised paper.

References: Some references are not discussed.

Thank you for suggesting these reference. We will incorporate a discussion of them into the revised paper.

Weak1: The generalizability to other datasets and domains could be further explored.

We appreciate your feedback regarding the generalizability of eMIGM. eMIGM performs effectively with both VAE (ImageNet 256x256) and DC-AE (ImageNet 512x512) tokenizers despite their distinct latent spaces. This adaptability highlights the transferability of our design principles across different data representations. We plan to explore eMIGM's application across diverse datasets and domains in future work.

Weak2: The paper could benefit from a more detailed discussion of the practical implications of using eMIGM

We appreciate this valuable suggestion. The practical implications of eMIGM are significant and multifaceted. Our model achieves higher-quality sample generation with fewer sampling steps (NFE), offering substantial advantages in computational resource utilization. Based on these efficiency gains, we believe eMIGM has strong potential for effective application across diverse domains currently served by diffusion models, including text to image and video generation. Besides, we systematically identify better design choices in masked generative modeling, offering practical guidance for developing efficient models. Our framework demonstrates how empirical analysis can effectively guide architectural decisions in this domain.

Q1: Are there any limitations or adjustments needed when applying the model to higher or lower resolution images?

Thank you for this question. When applying eMIGM to different resolutions, we find that the model architecture is inherently resolution-agnostic. The transformer backbone can process different sequence lengths without architectural modifications, allowing for flexible adaptation to various image resolutions. Our experiments on both 256×256 and 512×512 resolutions demonstrate this capability. The primary adjustment needed is simply retraining the model on the target resolution data, as is standard practice with most generative models.

Q2: What are the main computational bottlenecks when scaling up eMIGM models

Thank you for this important question. In our current implementation and experiments with eMIGM, we have not yet encountered significant computational bottlenecks that would require specialized optimization techniques. The model scales effectively with our available computational resources.

Q3: are there any plans to explore more efficient architectural designs?

We appreciate this valuable question. For future efficient architectural designs, we are exploring two promising directions: (1) U-ViT architecture, which achieves comparable performance to DiT on ImageNet 256x256 while using only ~30% of training FLOPs through its U-shaped design; and (2) REPA-inspired alignment between eMIGM's intermediate Transformer outputs and pretrained DINOv2 encoder features. Our preliminary experiments with the latter approach showed slight improvements when aligning earlier layers (4th), while later layers (18th) degraded performance. Though current benefits are marginal, more systematic investigation could yield substantial efficiency gains in future work.

Q4: Could you discuss the potential applications of eMIGM beyond image generation?

We appreciate this important question about eMIGM's broader applications. The model's core mechanism of predicting masked content from surrounding context naturally extends to various image manipulation tasks including inpainting (treating missing regions as masks), conditional editing (regenerating masked objects based on specified conditions), and outpainting (predicting exterior content around a central image). Beyond images, we believe eMIGM could be adapted for text to image, video, and audio generation, though this would require carefully designed input formats and potential architectural modifications to accommodate these diverse data modalities.

Experimental Design1: The claim of "CFG with Mask" and "fake class CFG"

Thank you for your question. In standard Classifier-Free Guidance (CFG) used in diffusion models and MAR, training involves occasionally replacing the class label with a dedicated 'fake class' token, which is distinct from the 'mask' token used for image patches. Our "CFG with Mask" approach instead uses the existing 'mask' token to replace class labels during training, eliminating the need for a separate 'fake class' token. We will clarify this distinction in the final paper.

Experimental Design2: Network architectural details come too late and What is the baseline performance of the discrete variant of eMIGM?

Thank you for your valuable suggestion. We appreciate your feedback and will incorporate more detailed network architectural information in an earlier section of our paper to enhance clarity. As briefly mentioned at the beginning of Section 4, our experiments exclusively focus on continuous masked-based generative models to mitigate the information loss associated with discrete tokenizers. Consequently, we did not develop a discrete variant of eMIGM, particularly since prior work has demonstrated that discrete variants of MAR perform significantly worse than their continuous counterparts. We will ensure all experimental details are presented more clearly in the final version of our paper.

Experimental Design3: The model steps of eMIGM.

We sincerely appreciate your valuable feedback. In response, we conducted an experiment on 512x512 image generation using eMIGM-L with NFE=64x1.25 (as presented in Table 3 of our paper). In this configuration, the diffusion model requires 14 sampling steps. To determine the additional computational cost of the diffusion model beyond the main transformer's NFEs, we measured the sampling speed with different diffusion steps. Our measurements on a single A100 GPU with batch size 256 show that the diffusion model introduces approximately 14% additional computational overhead beyond the main transformer's NFE requirements. Thank you for raising this concern about NFE comparisons between eMIGM and diffusion models. While eMIGM does include diffusion steps within each masked step, since transformer forward passes remain the primary bottleneck, NFE continues to be a valid efficiency metric. We will address this point more thoroughly in the revised paper, ensuring clarity and precision in our presentation.

Weakness1: The effectiveness of the proposed method, or the necessity of the unified masked diffusion modeling framework. We appreciate your valuable feedback.

Regarding the necessity of the unified masked diffusion modeling framework, it serves two primary purposes. First, it enhances the theoretical understanding of masked generation and simplifies comparisons between different methods (e.g., MAR vs. MaskGIT). Second, it facilitates a systematic exploration of the design space and allows for the integration of techniques from related areas. For example, advancements in diffusion models can be readily incorporated into our diffusion head training, helping to advance the state-of-the-art in masked image generation.

Regarding the effectiveness of eMIGM, our comprehensive experimental analysis demonstrates key advantages:

1. Through systematic exploration of the training and sampling design space, we identified critical factors influencing model performance and efficiency. These findings provide actionable insights for developing future masked generative models.
2. The proposed time-interval strategy for classifier-free guidance (CFG) empirically improves sampling efficiency while maintaining generation quality.

These improvements are substantiated by direct comparisons with MAR baselines. For ImageNet 512×512 generation, eMIGM achieves superior computational efficiency, requiring only 64×1.25 NFEs compared to MAR's 256×2 NFEs while maintaining competitive FID scores.

Q1: what makes the abbreviation "eMIGM"?

Thank you for your question. The abbreviation "eMIGM" stands for "Effective and Efficient Masked Image Generation Models.", where the prefix 'e' denotes both effectiveness and efficiency.

Q2: Why using continuous models like MAR instead of MaskGIT pipeline.

Thank you for this insightful question. We will clarify that our work focuses exclusively on continuous masked-based generative models. As briefly noted in Section 4, this choice stems from the need to avoid information loss inherent in discrete tokenizers. Previous research has consistently shown that discrete variants of MAR yield substantially inferior performance compared to continuous implementations. We will emphasize this choice more clearly in the final version of our paper and apologize for any previous lack of clarity.