ImageFolder: Autoregressive Image Generation with Folded Tokens

"Q5. CLIP v.s. DINO."

We would like to clarify that, as discussed in Line 301, we claimed that DINOv2's feature is not suitable for reconstruction as it loses pixel-level details. In our preliminary experiments, we compare DINO, CLIP (image encoder) and SAM as the distillation approach. As shown in Table B, DINO outperforms CLIP and SAM. We consider the necessity of an additional branch is to capture pixel-level details for reconstructing an image. Both the CLIP and DINO features are semantic-driven features that discard pixel-level details to boost understanding capability thus adopting CLIP features cannot alleviate the issue.

Table B. Extension of Table 2 with CLIP encoder.

---

"Q6. Tradeoff between token number and token length."

We apologize for the confusion. Both sentences (Line 408 and 340) aim to describe our core idea of reducing token length (in AR modeling) while keeping the token count. We have revised the expression in the revised version.

---

Q7. K=1 for the quantizer.

We would like to clarify that K=1 for the first scale is out of a generation consideration. The first scale with K=1 is not expected to have a satisfactory image quality. Instead, we hope it to be coarse and diverse enough to leave room for the subsequent steps to diversely enrich the details in a progressive manner as shown in Figure 7.

We tried to start residual quantization from K=2 however this will lead to an inferior generation quality, i.e. 3.53 (K=1) v.s. 3.92 (K=2) gFID.

---

"Q8. Add tokenizer comparison."

Please refer to the response to Weakness 1-2.

---

"Q9. 4-branch quantizer."

In the results reported in Table 6(c), the contrastive loss is only applied in one branch while keeping the other three branches free.

After submission DDL, we also tried a 3-branch setting, with two branches adding contrastive loss with two different pre-trained encoders and leaving one branch free. However, we still got >4 gFID. We did not observe a gFID gain with a branch number larger than 2.

---

We hope our response has addressed your concerns. Please feel free to reach out with any additional questions.

If you find our paper intriguing and promising, we would greatly appreciate it if you could consider raising your score to help increase the visibility of our work.

We thank the reviewer for the time and effort in reviewing our submission.

---

"W1-2. Lacks comparison to VAE and tokenizers. The VAR model training has not reached the final results."

We updated Table 4 in the revised submission to reflect the comparison with the previous tokenizers and demonstrate the final results of generation. Please note that the only change to the training recipe is that we adopt the stronger DINO discriminator used in the VAR tokenizer's training as shown in Table 3.

• Notably, our tokenizer achieves a 0.8 rFID which outperforms MAGVIT-V2 (0.9 rFID) as well as the tokenizer used in VAR (0.9 rFID).
• Our tokenizer achieves a 58.0 linear probing accuracy on the ImageNet val set, significantly outperforming 11.3 using VAR's tokenizer.
• When using tokens from ImageFolder, the same GPT2 architecture used in VAR can achieve a 2.6 gFID with a 10-step inference schedule which significantly outperforms the 3.3 of the original VAR.

---

"W3. Additional experiments on only the semantic branch or only the detail branch under the same setting. A single branch incorporates semantic information."

We thank the reviewer for the suggestions. We consider the experiments about the suggested only semantic branch and only detail branch are available in Table 3 as ID 4 (only detail) and ID 5 (only semantic). We combine Table 3 and report the additional experiment of a single branch with semantic information as shown in Table A. We can notice that two branches with one for semantic and one for detail can lead to the best performance. We have incorporated the results in Table 8 of the revised version.

Table A. Additional experiments about semantic settings.

---

"W4. Quantizer dropout"

We agree that quantizer dropout is not the core contribution of ImageFolder. Instead, we consider the major contribution of ImageFolder as (1) we explored the tradeoff between rFID and gFID and provided a solution to balance them, (2) we analyzed the impact of semantics to reconstruction and generation, and provided an approach to address both of semantics and details, and (3) extensive experiments and discussion towards a state-of-the-art image tokenizer (better rFID of 0.8 than MAGVIT-v2's 0.9 and VAR's 0.9) for generation.

We explicitly mentioned quantizer dropout in the method section since the training of quantizer dropout requires several model-specific settings to make it functional. We will discuss the details in the response to Q4.

---

"Q1. "tokens" in Table 1"

We would like to clarify that "tokens" in Table 1 refer to input tokens.

---

"Q2. $L_i$ in Line 247. Patch size $K=11$ in Table 3."

Thanks for the suggestion. We would like to clarify that $L$ and $K$ denote the input patch size and quantized token size. We apologize that $L_i$ in Line 247 is a typo which should be $K_i$. To avoid potential misunderstanding, we use "quantized token size" to describe $K$ in Table 3 in the revision.

---

"Q3. Understanding of Fig 5 (MSVQ)."

In Fig 5, the grid size should keep increasing while it is not necessary to be doubled. As discussed in Line 377, we used [1,1,2,3,3,4,5,6,8,11] in ImageFolder as the final setting.

We follow VAR to use multi-scale residual quantization (MSVQ) as our subquantizer. Specifically, MSVQ uses only 1 token to represent an image in the first residual layer and increases the token number with more residual depth. Combining quantizer dropout, different residual quantizer in MSVQ represents an image with different bitrates from low to high. This will lead to an image reconstruction from coarse to fine as shown in Fig 7. During the next-scale autoregressive prediction, the finer tokens are conditioned on the coarser tokens to gradually refine an image which permits a global understanding of the image compared to the traditional next-token prediction.

---

"Q4. Dropout."

Dropout is not necessary for a simple semantic quantizer but it can significantly improve the reconstruction and generation as demonstrated in Table 3. Empirically, we are not able to only dropout in one branch as it will lead to a severe stability issue and thus we made a note in Line 273. In addition, as denoted in Equ 3, the quantizer dropout should happen with a $N_{start}>3$ to ensure stability. We detailed those settings in the method section to enhance the reproducibility.

We thank the reviewer for the time and effort in reviewing our submission.

---

"W1/Q1. The training is based on LlamaGen. Scratch training?"

We would like to clarify that, as discussed in Line 373, our training hyperparameters/settings are based on the VQGAN training recipes of LlamaGen if no otherwise specification. However, the ImageFolder tokenizer is trained from scratch instead of loading the weights from LlamaGen.

---

"W2/Q2. Token length with scales [1, 1, 2, 3, 3, 4, 5, 6, 8, 11]."

We sincerely apologize for the typo. The token length for [1, 1, 2, 3, 3, 4, 5, 6, 8, 11] should be 286 = $1^2 + 2^2 + \cdots + 11^2$. The 265 is the token length for [1, 1, 2, 3, 3, 4, 5, 6, 8, 10]. We have updated the token length in the revised manuscript.

---

"W3/Q3. Ablation study on token length."

We report an ablation study on the token length. As shown in Table A, the token length of [1, 1, 2, 3, 3, 4, 5, 6, 8, 11] achieves the best tradeoff.

Table A. Ablation study on the token length.

We also did more experiments to investigate the scale setting for the first several tokens but we noticed that the scale settings used in VAR, i.e. [1,2,3,4,5,6,7,8,10,13,16] is a good practice. And our setting generally follows their token number for each scale. For example, VAR has 256=16x16 tokens and ImageFolder has 242=11x11x2 tokens in the last scale.

---

We hope our response has satisfactorily addressed your concerns. Should you have any further questions, please do not hesitate to reach out.

We thank the reviewer for the time and effort in reviewing our submission.

---

"W1. Performance indicators in the article are not in place, e.g. Table 1, 2"

We thank the reviewer for the suggestion. We have updated the description of performance indicators in the revised version.

---

"W2. Several hyperparameters and settings are missing for Equ 4."

We thank the reviewer for the suggestion. As discussed in Line 373, we followed the VQGAN training settings in LlamaGen for those hyperparameters. In the revised version, we explicitly provided a detailed description of hyperparameters for Equ 4.

---

We hope our response can address your concerns. Your feedback has been constructive, and we sincerely thank you for your time and effort.

"Question 1. Comparison against other tokenizers. Codebook utilization rate."

We updated Table 4 to include additional comparisons against other tokenizers. ImageFolder achieves the best performance against previous counterparts in terms of both rFID (0.8) and linear probing accuracy (58.0), including MAGVIT-v2 (0.9 rFID) and VAR's tokenizer (0.9 rFID). Please note that the linear probing is conducted on the quantized tokens instead of intermediate layers in the generator.

Table A. Codebook utilization of different codebook sizes of ImageFolder tokenizer.

As shown in Table A, we would like to clarify that the codebook utilization rate of ImageFolder is 100% among multiple codebook sizes.

---

"Question 2. Performance on other generative schemes."

We thank the reviewer for raising the generalization request of ImageFolder. A recent exploration of a continuous variant of ImageFolder (without hard quantization) demonstrates remarkable success in diffusion-based models as well.

Table B. Comparison against Diffusion-based models. IC: ImageFolder (continuous variant).

As shown in Table B, we compare ImageFolder (continuous) + SiT-XL [A] against the latest concurrent work REPA [B] + SiT-XL on image generation on ImageNet 256x256 following the diffusion-based convention. We can notice that the ImageFolder can achieve an even better performance than the state-of-the-art diffusion-based approach. This further demonstrates that the ImageFolder's semantic-rich latent representation is vital for image generation regardless of diffusion or autoregressive.

[A] SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers, ECCV 2024

[B] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, Arxiv Oct 24

---

Please do not hesitate to let us know if you have any further questions. We hope our response can address your concerns and we would greatly appreciate it if you could increase your rating.

We thank the reviewer for the time and effort in reviewing our paper.

---

"Weakness 1. Some components have been explored before, such as semantic regularization, multi-scale RQ and parallel decoding."

We respectfully disagree with the claim. There exists some earlier works that share similar concepts, however, completely different in design. Our approach made fundamental improvements compared to previous methods.

• As discussed in Line 295-305, a concurrent work VAIL-U (Arxiv, Sept 24) adopted a semantic constraint targeting understanding capability. However, VALI-U's semantic injection leads to a severe performance drop, i.e., -0.5 rFID and -1.2 gFID. Remarkably, our semantic regularization with an additional branch for detail information leads to a 1.67 rFID and 3.03 gFID gain. We also provided an in-depth analysis discussing the properties of semantic regularization to explain the gain and difference against the previous approach in Line 276-305.

• As discussed in Line 419-426, a baseline approach VAR (NeurIPS 24) proposed multi-scale RQ. However, VAR's token length is significantly large due to the RQ, i.e. 680 tokens. In imagefolder, with the spatial-preserving Q-Former and Product quantization, we successfully reduced the MSVQ's token length to 286 with an even better gFID of 2.6 compared to 3.3 of VAR.

• We did not recognize a work that leverages parallel decoding that is similar to our approach in AR-based image generation. It will be helpful if the reviewer can introduce us to specific papers and we are happy to provide detailed comparison and discussion.

Beyond the above difference, we consider the major contribution of ImageFolder is (1) we explored the tradeoff between rFID and gFID regarding token length, and provided a solution to balance them, (2) we analyzed the impact of semantics on reconstruction and generation and provided an approach to address both of semantics and details and (3) extensive experiments and discussion towards a state-of-the-art image tokenizer (superior rFID and gFID) for generation.

In all, we believe our work provides significant insights and improvements compared to previous methods.

---

"Weakness 2. Discussion about product quantization (PQ). More visualization of Fig 8."

We would like to clarify that we have provided additional visualization similar to Fig 8 in the Appendix (Fig 12). In Lines 203-214, we have provided a detailed formulation of PQ and included sufficient references for readers to understand the concept.

In addition, we would like to clarify that we did not claim tokens have a "disentangled nature" primarily due to PQ (though some audio tokenizer designs, such as Wav2Vec, do have). Instead, we consider the information encoded in each branch to be determined by semantic regularization. As discussed in Line 295-305, one branch is designed to align DINO features. However, since the DINO feature discards high-frequency detail information of images (Table 2), with a reconstruction objective, another branch will capture the complementary detail information. To further facilitate the understanding of PQ, we provided more discussion about the product-quantized space in the revision in the Appendix to enhance the readability.

---

"Weakness 3. Missing references, e.g. PQ in audio tokenization."

We thank the reviewer for introducing us to the related work. We have cited [1] in the revised manuscript.

We would like to further clarify that the mentioned work is a concurrent work published on Arxiv in June 2024, within 4 months of the ICLR ddl according to guidelines. In addition, [1] aims to tackle the low codebook utilization rate in audio tokenization and successfully increase the codebook utilization from 13.5% to 97.5%.

However, the codebook utilization rate of ImageFolder (including all ablations in Tab 3) is 100%. PQ is not the reason why we achieve 100% codebook utilization. Even with just a single branch, we can still achieve 100% utilization. Instead, the introduction of PQ in ImageFolder aims to (1) provide an additional branch for encoding both detail and semantic information, (2) increase the potential combination number in image representation, and (3) shorten the token length.

In this way, we consider our work to have fundamental differences compared to [1].