Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective

We appreciate your constructive feedback and would like to clarify several points to enhance the understanding of our contributions.

1. Regarding the analysis of AE and PCA and the claim of the observation

We want to emphasize that the analysis of AE and PCA is not the core focus of our theoretical framework. We mention it to provide intuitive insights into the latent space induced by VQ autoencoder and our proposed discriminative tokenizer.

While we acknowledge that the distinction between reconstruction and generation has been explored in prior works, our primary contribution lies in offering a theoretically unified perspective on this issue and proposing a novel method to disentangle the encoder and decoder processes for a more stable latent space, which is neglected in previous work.

2. Regarding the stability

To further evaluate the stability of latent space, we analyze the Negative Log-Likelihood loss for AR models associated with different tokenizers. Our results highlight the nondeterministic nature of the latent space, showing an NLL loss of 3.347 for the discriminative tokenizer versus 7.104 for the VQ tokenizer, where a lower NLL loss indicates a more stable latent space, underscoring the efficacy of our approach. The resulting FID score also demonstrates that stable latent space can improve the performance of AR models.

3. The FID score of VQGAN

We adopt the VQ tokenizer used in the MaskGIT mdoel, which is an improved version and performs better than the one used in the original VQGAN. We reproduced the class-unconditional results by ourselves, yielding an FID score of 24.38, which is reasonably within range and slightly worse than MaskGIT's score of 20.7. Moreover, we check the experiments in the original VQGAN paper and it only presents the results of class-conditional image generation results (FID 15.78), which is not the same experimental results as above and should not be directly compared. We will clarify these experimental settings in the revised paper.

4. Regarding the pretrained SSL encoder in our method

We recognize the dataset difference used in SSL encoders and have included an experimental comparison of various SSL encoders in Fig 3(a). We adopt three SSL models and all of them are trained only on ImageNet, which is the same as the VQ model. As detailed in section 4.5, the small size autoregressive model trained with tokenizer induced by the iBOT SSL model, which shares a similar learning objective with DINOv2 (patch level discriminative objective), already achieves superior performance compared to a much larger model trained with VQ tokenizer. The comparison demonstrates the efficacy of the discriminative tokenizer.

We adopt a more powerful SSL encoder DINOv2 to demonstrate that the discriminative tokenizer with more powerful SSL representations can boost the performance of auto-regressive generative models by a large margin, which is not observed in existing AR + VQ models, highlighting the main limitation of image auto-regressive models is the encoder used for latent space construction.

5. Sample Cases

The FID and IS scores are the established metrics for image-generative models. The cases shown in the paper are just for reference. We can provide more samples in the revised pdf.

6. Regarding the baselines

There exist too many diffusion variants and most of them are based on the LDM or DiT architecture, therefore we opt to focus on foundational works, such as LDM and DiT, as baselines. Furthermore, our proposed method is built upon a different theoretical framework, aimed primarily at surpassing widely accepted baseline models.

7. The logical transitions and question on stability

In section 2.1, we first conduct a theoretical analysis that demonstrates the necessity of considering both $\mathcal{D}^{H}$ and $\mathcal{D}^{G}$ simultaneously in the latent space for generative models. Current VQ-based latent generative models represent one pathway for constructing latent space by optimizing $\mathcal{D}^{H}$, an autoencoder objective. In contrast, our approach focuses on optimizing the latent space with $\mathcal{D}^{G}$ to learn lower-dimensional separable features for generative models, which is an SSL objective. Following this, in section 2.2, we explore the differences in latent space induced by various learning objectives and find that the success of combining iterative generative models with VQ latent space is attributable to the stabilization ability of the iterative decoding strategy. In contrast, our proposed discriminative tokenizer directly stabilizes the latent space, which can naturally benefit auto-regressive models. Therefore, the above analysis and findings provide the foundation for the proposed method: SSL + Kmeans.

8. Question on Linear probing

The linear probing is done on the hidden states of auto-regressive models, which is the same as the evaluation protocol of iGPT and VIM.

9. Question on limitation

As discussed in the paper, the latent generative models are affected by both the encoder and decoder. Our strategy to disentangle these components aims to strengthen the encoder’s capabilities with the integration of pre-trained SSL methods unsupervised. We use the VQGAN model as the decoder to compare with existing VQ-based models. Although improvements to both the encoder and decoder could yield further benefits, our findings indicate that the encoder's ability is currently the primary limiting factor for autoregressive models since the rFID score of VQGAN is low enough. Improvements using SSL encoders and K-means considerably elevate performance, as demonstrated in Table 2, Fig 2(a), and Fig 3(a).

We kindly ask you to reconsider the evaluation of our work in light of these clarifications. We believe that our research contributes significantly to the understanding and development of image tokenizer and latent autoregressive models.

Thank you for your thoughtful consideration of the paper and your constructive feedback.

1. Factual Clarification

We respectfully disagree with the points raised and would like to clarify our positions.

1.1 Regarding the MIM models like MaskGIT and MAGE, as well as diffusion models, it is important to note that they can indeed revise the predicted tokens from previous iterations. As outlined in Section 2.2, the decoding mechanism of iterative generative models can be expressed as:

$

p(x^T)=\prod^{T}_{i=1} p(x^i|x^{i-1}),

$

where $x^{i}$ represents the predicted tokens (the entire image) in the i-th iteration. In MIM models, tokens with low probability (from the softmax of logits) are replaced with UNK tokens and re-generated in subsequent iterations. For diffusion models, the core mechanism involves iterative denoising, which is an established concept, where predicted tokens in one iteration are further modified and re-generated in the next iteration. Thus, we believe our statement in the paper is accurate and we encourage the reviewer to reconsider the interpretation in light of this clarification.

1.2 Regarding the Parti model, it is important to emphasize that it is a VQ-based text-to-image generation model that benefits from extensive training data and a large model size. However, it would be misleading to assert that Parti demonstrates success in image auto-regressive generative models. When we talk about GPT model, we are usually interested in the learning efficiency from data rather than merely enlarging dataset size and model size. By removing the influence of dataset size and model size, we note that the core architecture of Parti, which is a VQ-based AR model, does not perform as well compared to other generative models, limiting its scalability [2]. Our research specifically targets autoregressive image generative models operating without text guidance, evaluated on the ImageNet benchmark under rigorous experimental settings. In addition, methods later than Parti like MagVITv2 [1] and VAR [2] also claim to be the "first" language models outperforming diffusion models, yet they do not qualify as genuine autoregressive models. In contrast, our proposed DiGIT model is based on a pure GPT architecture, without any modifications to the decoding strategy. Therefore, we do not agree with the assertion that "the Parti paper has already demonstrated that image autoregressive models have similar scalability to GPT".

2. The statement of "common misconception"

We cited relevant work [1] in Line 43 to support the statement regarding the "common misconception." This is a well-recognized aspect within the research community, as pointed out by reviewer K39P as well. The "optimal latent space for reconstruction" refers to the latent space achieved by an autoencoder model that can yield the lowest rFID, which is a straightforward statement that does not require further elaboration.

3. The CFG in the experiment

We choose not to employ CFG as a default method due to its tendency to sacrifice diversity in generated images. Instead, we compare all models using class labels as conditions. It is worth noting that autoregressive models can benefit a lot from CFG [3] as well. However, autoregressive models are renowned for their prompt engineering capabilities and our philosophy is to revive the GPT model with as few modifications as possible. Therefore, we intentionally do not adopt CFG as the default method for all the models in our experiments.

4. The table in the paper

We attempted to consolidate the tables in the submission version but exceeded the page limit. In the camera-ready version, one additional page will be allowed and we could improve the table presentation if accepted.

After addressing these potential misunderstandings of the paper, we kindly request a reevaluation of our paper.

[1] Lijun Yu, Jos’e Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David C. Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to visual generation. ArXiv, abs/2310.05737, 2023.

[2] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. 2024.

[3] Sun, Peize and Jiang, Yi and Chen, Shoufa and Zhang, Shilong and Peng, Bingyue and Luo, Ping and Yuan, Zehuan. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv:2406.06525

Dear Reviewer Sn2s,

I hope this message finds you well. The discussion period is ending soon, I am writing to emphasize the importance of your review for our submission. Your score is significantly lower than the other three reviewers, and we believe this discrepancy may indicate a misunderstanding or oversight.

We have addressed all the concerns in our detailed rebuttal and would appreciate your prompt attention to it. A thorough reassessment is crucial to ensure a fair evaluation.

Your expertise is highly valued, and we trust that a reconsidered review will reflect the true merit of our work.

Thank you for your immediate attention to this matter.

Best regards, Authors

Thank you for your thoughtful consideration of the paper and your constructive feedback.

1. Performance of the Proposed Method on Image Reconstruction

We conduct an experiment to assess the reconstruction performance of the proposed discriminative tokenizer. We use the golden tokens obtained from the discriminative tokenizer to reconstruct the images and calculate the rFID score, which yields a result of 1.92. In comparison, the rFID score for the corresponding VQ tokenizer is 1.67. This indicates that the impact of our proposed tokenizer on reconstruction quality is minimal. We appreciate your suggestions and will include these reconstruction results in Table 2 of the revised paper.

2. Regarding the Stability of the Proposed Method

(1) Stability of Latent Space Induced by Tokenizers

We quantitatively measure the stability of the latent space in Table 1, incorporating metrics such as the rate of token changes and cosine similarity when varying levels of noise are introduced. The results align with intuition well that SSL models demonstrate greater stability against noise, as they learn high-level semantic features of images rather than low-level appearance features as in traditional autoencoders.

(2) Stability of Autoregressive Models

To evaluate the stability of autoregressive models, we compare the negative log-likelihood (NLL) loss of models using different tokenizers, thereby demonstrating the nondeterministic nature of the latent space. We monitor the NLL loss values for both the discriminative tokenizer and VQ tokenizer, both using a vocabulary size of 1024 and the same model size: 3.347 for the discriminative tokenizer versus 7.104 for the VQ tokenizer. A lower NLL loss indicates a more stable latent space for autoregressive models. We will include the loss curve comparison in the revised version.

3. Connection Between the Theoretical Analysis and the Proposed Algorithm

In section 2.1, we first conduct a theoretical analysis that demonstrates the necessity of considering both $\mathcal{D}^{H}$ and $\mathcal{D}^{G}$ simultaneously in the latent space for generative models. Current VQ-based latent generative models represent one pathway for constructing latent space by optimizing $\mathcal{D}^{H}$, an autoencoder objective. In contrast, our approach focuses on optimizing the latent space with $\mathcal{D}^{G}$, which is a SSL objective. Following this, in section 2.2, we explore the differences in latent space induced by various learning objectives and find that the success of combining iterative generative models with VQ latent space is attributable to the stabilization ability of the iterative decoding strategy. In contrast, our proposed discriminative tokenizer directly stabilizes the latent space, which can naturally benefit auto-regressive models. Therefore, the above analysis and findings provide the foundation for the proposed method: SSL + Kmeans.

Dear Reviewer eGbB,

I hope this message finds you well. We are grateful for your valuable feedback on our submission and are pleased to see your positive score. In our responses, we have addressed the points you raised in detail.

As the discussion period is coming to a close soon, we kindly ask if you could review our responses at your earliest convenience. We are eager to know if our explanations have alleviated your concerns. If there are still areas needing improvement, your insights would be greatly appreciated and instrumental in enhancing our work.

Thank you once again for your thoughtful review and support.

Warm regards, Authors

Thank you for your thoughtful consideration of the paper and your constructive feedback.

1. Motivation for Adopting a Self-Supervised Model as Encoder/Tokenizer

In section 2.1, we first conduct a theoretical analysis that demonstrates the necessity of considering both $\mathcal{D}^{H}$ and $\mathcal{D}^{G}$ simultaneously in the latent space for generative models. Current VQ-based latent generative models represent one pathway for constructing latent space by optimizing $\mathcal{D}^{H}$, an autoencoder objective. In contrast, our approach focuses on optimizing the latent space with $\mathcal{D}^{G}$, which is a SSL objective. Following this, in section 2.2, we explore the differences in latent space induced by various learning objectives and find that the success of combining iterative generative models with VQ latent space is attributable to the stabilization ability of the iterative decoding strategy. In contrast, our proposed discriminative tokenizer directly stabilizes the latent space, which can naturally benefit auto-regressive models. Therefore, the above analysis and findings provide the foundation for the proposed method: SSL + Kmeans.

2. Clarification of "Stability" and "Learnability" of the Latent Space

We measure the stability of the latent space quantitatively in Table 1, incorporating metrics such as the rate of changed tokens and cosine similarity when varying levels of noise are introduced. It intuitively validates that SSL models exhibit greater noise stability because they learn high-level semantic features of images, rather than low-level appearance features as in traditional autoencoders.

We plan to include the negative likelihood loss as a measure of learnability in our revised version (3.347 for the discriminative tokenizer compared to 7.104 for the VQ tokenizer, using the same vocabulary size and model size). The lower NLL loss represents more learnable latent space.

3. Addressing the Argument Regarding Iterative Decoding

The iterative decoding strategy serves as the foundation for iterative generative models, including Masked Image Modeling (MIM) and diffusion models. For MIM [1], low-density tokens are revised during the iterative decoding process. Similarly, in diffusion models [2], low-density data undergo perturbation through Gaussian noise and are subsequently denoised in an iterative manner.

[1] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman. MaskGIT: Masked Generative Image Transformer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2022.

[2] Y. Song, S. Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances in Neural Information Processing Systems, pp. 11895--11907. 2019.

4. The Decoder for Pixel Rendering

The decoder setup is elaborated upon in Section 4.3, where we detail the additional training of a decoder for pixel rendering. To showcase the generalization and robustness of our discriminative tokenizer, we employ both autoregressive models (VQGAN) and MIMs (MaskGIT) as decoders. The experimental results presented in Tables 2 and 3 indicate that the performance gap between AR and MIM decoders is minimal, demonstrating the efficacy of the discriminative tokenizer.

5. Clarification of the Metric

The metrics used to assess stabilization in Table 1 are the rate of token changes and the cosine similarity between perturbed and original tokens. We will ensure that these metrics are presented more clearly in the revised version.

6. Connection Between Our Method and Error Accumulation in Auto-Regressive Image Models

In Section 2.2, we address the potential for error accumulation due to unstable VQ tokenizers across all latent generative models, including diffusion models, MIMs, and AR models. While iterative models like diffusion models and MIMs can rectify errors through their iterative decoding strategies, autoregressive models do not possess this capability. Therefore, we propose a direct approach to stabilize the latent space for autoregressive models, effectively reducing errors caused by unstable tokenizers in the decoding process.