Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance

We appreciate your thorough review. We address your concerns and questions below. All visual results can be found in the submitted PDF. Please zoom in on the figure of the submitted PDF for the best view.

$\**Clarified visual comparison.**$

In Fig. 6 of the submitted PDF, we directly compare the sampled images using MaskGIT and the proposed method with the same random seed on ImageNet 256 and 512 resolutions. With the proposed guidance, fine-scale details are enhanced, generating higher-quality images. Please refer to the Global Response for a detailed description.

$\**Impact of guidance in lower resolution.**$

Since the proposed guidance utilizes semantic smoothing, the effectiveness of the guidance can be affected by the resolution of the dataset or input token length. We first note that the proposed guidance is built upon the pre-trained vector quantizer and MGMs. However, we cannot find publicly available vector quantizers and MGMs trained on ImageNet 128 resolution. We tried to train both VQGAN and MaskGIT, which have to be trained sequentially due to the dependency. However, within the limited time frame and computational budget, we are unable to generate reliable results from MaskGIT, generating output with very low visual quality, which shows an inception score of around 20.

Though the VQGAN and MaskGIT pretraining is not converged, we measure the performance improvement by attaching the proposed guidance to the MaskGIT with ImageNet 128 resolution. We measure the IS (larger is better) and FID (lower is better) with a sampling temperature of 2.0. The IS increases from 21.2 to 29.5, and FID decreases from 44.0 to 33.6. Although the hyperparameter search is limited, the FID improvement is significant (23%) compared to the improvement in 256 (50%) and 512 (36%) resolutions. We expect that the performance improvement at lower resolutions will be similar if a well-pretrained quantizer and MaskGIT are utilized and sampling hyperparameters are suitable.

$\**More intuition for auxiliary task and visualization.**$

Thank you for your recommendation, which helps us better illustrate the proposed auxiliary task. We thought it would be beneficial for all reviewers to share deeper insights into the underlying intuition and the visual results, so we provided the review in a global response. Thank you once again for your valuable recommendation!

$\**Choice of the sampling temperatures.**$

In Fig. 5 of the main manuscript, utilized sampling temperatures are listed below.

• 256: [3, 4, 5.5, 15, 20, 23, 25, 30, 40, 60, 80]
• 512: [6, 10, 15, 20, 25, 30, 35, 40, 45, 50, 70, 90]

The sampling process of MGMs is known to be sensitive to the choice of various sampling hyperparameters [1]. Sampling temperature plays a crucial role in resolving the multi-modality problem, as explained in section 2. Thus, moderate sampling temperature can increase the sample quality and diversity, simultaneously. After some point, when the temperature increases, diversity increases while quality deteriorates. This is because the large sampling temperature gives strong randomness for re-masking of MGMs ($p(m_{t−1}|m_t, \\hat{x}_{0,t}$) in line 99), potentially masking relatively accurate tokens while leaving unrealistic tokens unmasked. As a result, the sampling of MGMs is sensitive to the sampling hyperparameters. We believe that our extensive ablation experiment can provide valuable insights for the community.

$\**Impact of the strong guidance and high sampling temperature.**$

In Fig. 11 of the submitted PDF, we compare the samples with our default config, with strong guidance (s=5) and high sampling temperature ($\\tau$=50). We found that the strong guidance often leads to undesirably highlighted fine-scale details (left column in Fig. 11 b) or saturated colors (right column in Fig. 11 b), similar to the large scale of CFG. High temperatures often lead to the collapse of the overall structure. As an extension of Fig. 5 a of the main manuscript, we further plot the FID and IS curves using larger guidance scales in Fig. 6 d. Using a large guidance scale decreases fidelity and diversity due to the phenomenon we mentioned above. We will add the analysis and figure in the final manuscript for a thorough investigation and to give further insight into the proposed guidance.

$\**Further clarification.**$

The $\\mathbf{m}\_{i}$ in eq. 7 denotes whether the $i$th input token $z\_t$ is masked or not. $\\mathbf{m}\_i=0$ if the $i$th input token is masked and $\\mathbf{m}\_i=1$ if not. Since the masked index of $z\_t$ and $x\_t$ are identical, the $\\mathbf{m}\_{i}$ in eq. 7 equals to the $\\mathbf{m}\_i$ in eq. 1. We will clarify this in the final manuscript.

---

[1] Chang, Huiwen, et al. "Muse: Text-to-image generation via masked generative transformers." arXiv preprint arXiv:2301.00704 (2023).

Thank you for your review. We address your concerns below. All visual results can be found in the submitted PDF. Please zoom in on the figure in PDF for the best view.

Clarifications of implementation details, figures, and code release.

Thank you for your suggestion to make our research clearer and more reliable. We will clarify and magnify the figures in the final manuscript. We will add more visual results for a clear comparison of the proposed method to the base MaskGIT model, as illustrated in Fig. 6 of the submitted PDF. Please refer to the Global Response for a detailed description.

The proposed method consists of two parts: MaskGIT and TOAST. Since we have not changed the MaskGIT architecture, we briefly explain the implementation details of TOAST below. The TOAST modules consist of three parts: token selection module, channel selection module, and linear feed-forward networks.

• (i) The token selection module selects the task or class-relevant tokens by measuring the similarity with the learnable anchor vector $\\xi_c$. We generate class conditional anchor $\\xi_c$ with simple class conditional MLPs.

• (ii) The channel selection is applied with learnable linear transformation matrix P. Then, the output of the token & channel selection module is calculated via $z_i = P \\cdot sim(z_i,\xi_c) \\cdot z_i$, where $z_i$ denotes the $i$th input token.

• (iii) After the feature selection, the output is processed with $L$ layer MLP layers, where $L$ is equal to the number of Transformer’s layers. The output of the $l$th layer of MLP blocks is added to the value matrix of the attention block in $(L-l)$th Transformer layer (top-down attention steering). Following the previous work [1], we add variational loss to regularize the top-down feedback path. A more detailed process and theoretical background can be found in [1].

Most of the implementation can be found in the official TOAST repository. Since our work is also based on the public repository, as mentioned in the main manuscript, we believe that the proposed guidance can be easily implemented and applied in various code bases. We will add the implementation details in the final manuscript for understandability and reproducibility. We also plan to release the source code as soon as we complete the documentation.

Measuring the computational efficiency.

We measure the computational efficiency by calculating the sampling time on A6000, where the batch size is set to 50. In Fig. 9 of the submitted PDF, we plot sampling time with FID/IS values compared to diffusion-based methods and various MGMs. Note that we exclude Token-Critic and DPC since the code is not publicly available. Though each model has slightly different implementation details for the architecture, an order of magnitude smaller NFEs of MGMs (see Table 1 of the main manuscript) lead to a significantly efficient sampling process compared to diffusion-based methods. With the proposed self-guidance, the proposed method surpasses the diffusion-based approaches and MaskGIT in both sampling efficiency and performance.

Generalization to text-to-image method or image-to-image translation.

The proposed method can be generally adopted for various tasks of MGMs. We briefly explain how the proposed method can be extended to various tasks.

• Text-to-Image generation (T2I): Because the proposed guidance can be generally defined with text condition, the proposed method can be easily extended to the T2I generation by replacing the class conditional module in TOAST with the text conditional module. Except for the architecture, the training and sampling strategies illustrated in Fig. 2 of the main manuscript can be adopted without modification for text embedding.

• Image-to-Image generation (I2I): For Image-to-Image generation, such as style transfer, the proposed method can be adopted to further enhance the sample quality. For instance, MaskSketch [2] utilizes pretrained MaskGIT to generate images from the sketch input. The proposed method fine-tuned on the target dataset can be additionally attached for the enhanced sample quality.

Choice of the quantizer.

Similar to continuous diffusion, the (discrete) latent space brings several strengths, such as semantic richness and computational efficiency. As a result, modern MGMs and various discrete domain generative image models mostly utilize vector-quantize-based encoder-decoders such as VQGAN and VQVAE. In this regime, our work is built upon the MGMs, which operate on VQ space. The latent space of various VQ-based encoders shows similar characteristics, such as rich semantic information, and lacks continuous semantic structure in contrast to pixel space. Furthermore, the proposed guidance and the auxiliary task can be generally defined in any discrete space. As a result, we expect that the proposed guidance will not be tied to the specific quantizer architecture and that the performance improvement will be similar.

---

[1] Shi, Baifeng, et al. "TOAST: Transfer Learning via Top-Down Attention Steering."

[2] Bashkirova, Dina, et al. "Masksketch: Unpaired structure-guided masked image generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Thank you for your comprehensive review and for identifying the ambiguous aspects of our experimental designs. We address your concerns and questions below. All visual results can be found in the submitted PDF. Please zoom in on the figure of the submitted PDF for the best view.

Include more visual results for clarification.

To clearly show the improvement with the proposed guidance compared to MaskGIT, we add more visual results in Fig. 6 of the submitted PDF. Please refer to Global Response for a detailed description. We also visualize the impact of the different sampling hyperparameters in Fig. 11 of the submitted PDF. The high guidance scale may deteriorate the sample quality by overly enhancing fine details, and high sampling temperature often leads the overall structure to collapse.

Using another model for the auxiliary task.

Although we argued that TOAST could be efficiently and naturally adopted to learn auxiliary tasks, other PEFT methods can also be adapted to generate guidance similarly. Before validating, we note that the benefits of using TOAST are twofold:

1. MGMs exhibit training bias of only predicting the [MASK] tokens while treating other input tokens as ground truth. As a result, MGMs tend not to correct errors in the input tokens. We resolve this by simply masking all tokens of the 2nd stage of TOAST prediction.

2. Since the output of the 1st stage of TOAST is directly reused to sample $p_\theta(\hat x_{0,t})$, it brings more efficiency for sampling (see Fig. 2 of the main manuscript).

We experiment with the prompt tuning-based approach with the proposed auxiliary loss. The prompt tuning shows reasonable performance for transfer learning of MaskGIT [1]. The prompt token length is set to 128 following [1], and the batch size is reduced to 128 due to the memory capacity. We first forward the $z_t$ with prompt tokens to obtain ${\mathcal{H}}(z_t)$ in eq.7. Then, we forward the MaskGIT with the ${\\mathcal{H}}(z_t)$ to calculate the ${\\mathcal{L}}_{aux}$ in eq.7. To train the model, we directly input the ${\\mathcal{H}}(z_t)$ after the embedding layer of MaskGIT.

We set the guidance scale to 0.2 since we manually found the result guidance was too strong. The result in the table below and Fig. 8 of the submitted PDF show that the improvement is marginal compared to ours. We suspect that the naïve prompt tuning does not suitably deal with the error correction due to the bias of MGMs (see loss curve in Fig. 8). Furthermore, it requires 3 models forward for each sampling step. With the above discussion and experiments, as well as the discussion in the main manuscript, we verify that the TOAST is a more suitable approach compared to the other PEFT methods in terms of performance and efficiency. We expect that a more meticulously designed model for the proposed guidance can further improve efficiency and performance and leave it for future work.

Explanations for the choice of hyperparameter (sampling step).

In our preliminary research on the impact of sampling steps ($T$), we mainly investigated $T$ around $\sim$18, which is commonly adopted by prior works. We further explore the larger $T$ in Fig. 10 of the submitted PDF using $T$=24 and 36 with sampling temperature $\\tau$=35 and 60, respectively. The results indicate that using $T$, larger than 18, does not ensure a performance increase, as the metrics either saturate or deteriorate despite higher computational costs.

Similar to findings in MaskGIT, where the optimal performance-efficiency trade-off is observed around $T$=8, our method shows a "sweet spot" around $T$=18. Up to this point, both quality and diversity increase as sampling timesteps and computational costs increase while outperforming MaskGIT with similar NFEs. This demonstrates that the proposed method shows more scalable and efficient results compared to the MaskGIT. We also found a similar phenomenon for experiments on ImageNet 512x512, and the optimal timestep is around 12~18 steps. In this context, we opt for the sampling timestep of 12 and 18 for the results. We will add the above experimental results and our analysis in the final manuscript to give a thorough insight into choosing the sampling steps.

Generalization to more complex datasets.

Thank you for your valuable comments. Since the suggested datasets are primarily utilized for text-to-image (T2I) generation tasks, pretrained VQGAN and MaskGIT are not publicly available.

Instead, the T2I MGMs, MUSE [2], can be utilized to demonstrate the generalization. Though the main concern of the paper is to improve the generative capabilities of MGMs in class conditional image generation, the proposed guidance and utilized fine-tuning architecture can be extended to T2I generation. Given that the only difference is the replacement of the class condition with a text condition, we can simply substitute the class conditional module in TOAST with a text conditional module while maintaining other fine-tuning and sampling strategies. (Please refer to the response for Lc5P for a detailed TOAST implementation)

However, within the limited time frame, it is challenging for us to find reliable code (since the official MUSE is not publicly available), preprocess the large datasets, and ablate the generalizability of the proposed guidance. Therefore, we leave the generalization performance on T2I as future work.

---

[1] Sohn, Kihyuk, et al. "Visual prompt tuning for generative transfer learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Chang, Huiwen, et al. "Muse: Text-to-image generation via masked generative transformers." arXiv preprint arXiv:2301.00704 (2023).