Reviewer 4

We appreciate the constructive comments from reviewer DwYd . We provide responses to each individual question below.

W1: The abstract doesn’t describe the full content presented in the paper.

We have updated the abstract to briefly summarize all four contributions outlined in Lines 59–66.

We like to emphasize that our main contribution in this work is discovery of token representation shrinkage phenomenon which was unknown to the community and is important for diversity of outputs of generative models. In addition, token representation shrinkage is fundamentally different from codebook collapsing phenomenon (which we will clarify in the W2 section). We have also adjusted the abstract to emphasize this.

W2: What’s the difference of code collapse and token representation shrinkage

These two phenomena are distinct issues in VQ and are fundamentally different. Codebook collapse refers to the deactivation of codes, while token representation shrinkage is the overall code distribution becomes narrower and fails to match the original embedding distribution regardless of code activation. Codebook collapse, as reported in many previous studies[1, 2, 3, 4], refers to the phenomenon where a portion of the codebook entries become "dead" (they are never or rarely used during training). In contrast, token representation shrinkage can occur even when all codebook entries remain active. Token representation shrinkage, which we first formally identified in our manuscript, refers to a situation where the learned token embeddings crowd into a small region of the latent space, thereby limiting representational diversity. Identifying the token representation shrinkage phenomenon is the major contribution in our study as it informs tokenizer design and behavior analysis of generative models.

W3: No (c) and (e) in Fig. 3. It is also unclear how the plot (b) w/o shrinkage was obtained.

We sincerely apologize for the typographical error. We have revised the figure reference to Fig. 3(a) accordingly in the updated manuscript.

Regarding Fig. 3(b) ("w/o shrinkage"), this plot was obtained by initializing the codebook based on a trained encoder, as described in Lines 153–158. Since the pretrained encoder produces well-dispersed embeddings with semantic separability, it helps mitigate token representation shrinkage by promoting a more balanced token distribution early in training.

W4: No quantitative comparison to previous works also aiming at improving VQ training.

We appreciate the suggestion to include comparisons with prior work on improving VQ training

We conducted additional experiments comparing the pretraining method with SimVQ 4, targeting codebook collapse.

As shown in Table 1, on CIFAR-10 with varying codebook sizes, our method achieves comparable reconstruction performance to SimVQ. And if we combine our method with SimVQ, it further improves the performance. That demonstrates that token representation shrinkage and code collapse are different problems. Addressing both of code collaspe and token representation will further improve the performance of vector quantization.

Table 1. Ablation on CIFAR-10 across different codebook sizes. MSE (×10^-4)

On the ImageNet-100 dataset (Table 2), our method also demonstrates better reconstruction quality, highlighting the effectiveness of our strategy in relative large-scale scenarios.

Due to resource limitation and time limitation, the generative results on Imagenet-100 have not finished yet. The initial training loss in the first few epochs are shown in Table. 3.

Table 2. VAR tokenizer performance on Imagenet-100.

Table 3. Training Loss(Cross Entropy) for VAR

Besides CIFAR-10 and Imagenet-100, we also conduct experiments on ODIR and Oxford Flower 102. The results as shown in Table 4 to Table 7. We notice that our simple solution improves the diversity of generative model on different dataset. It is important to emphasize that the major contribution of this paper is the discovery of the phenomenon of token representation and understand how it affect the diversity of generative models’ outputs. We use a simplest method of pretraining encoder to show the effects. But the encoder pretraining method is not our major contribution. Therefore, it won’t be surprising to see advanced method such as SimVQ show better enhancement of diversity.

Table 4. VAR tokenizer performance on ODIR.

Table 5. VAR generation results on ODIR

Table 6. VAR tokenizer performance on Oxford Flowers 102. Cos Dis represents Cosine distance in codebook. Euc Dis represents Euclidean distance of codebook.

Table 7. VAR tokenizer performance on Oxford Flowers 102

W5: The experimental setting focuses on relatively small dataset (CIFAR=10 and ImageNet-100)

We are currently conducting experiments on 256 × 256 resolution for Imagenet-1k. The preliminary results of the first few epoch are listed in Table 8 and 9. We come from an academic institution and only have access for 2-4 GPUs. Thus, we are still running the experiments can not obtain the final results in time.

Table. 8 LPIPS(×10^-1) during training on Imagenet-1k

Table. 9 MSE(×10^-2) during training on Imagenet-1k

We sincerely thank the reviewer for highlighting this important direction. We plan to extend our study to ImageNet-1K and potentially other real-world scenarios and will include this in the final version of the paper.

W6: add recall and precision metrics.

We appreciate the insightful feedback from the reviewer. We have included Precision and Recall scores as measurement in the main text and attached as in Table 10, Table 11 and Table 12. VQ with reduced token representation shrinkage shows higher recall in generate outputs in VAR for ODIR and MaskGIT for ImageNet dataset.

Table. 10 VAR generative results on ODIR

Table. 11 MaskGIT generative results on Imagenet-100

Table 12 VAR generative results on Imagenet-100

References

[1] Yu, Jiahui, et al. "Vector-quantized image modeling with improved vqgan." arXiv preprint arXiv:2110.04627 (2021).

[2] Zheng, Chuanxia, and Andrea Vedaldi. "Online clustered codebook." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Huh, Minyoung, et al. "Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks." International Conference on Machine Learning. PMLR, 2023.

[4] Zhu, Yongxin, et al. "Addressing representation collapse in vector quantized models with one linear layer." arXiv preprint arXiv:2411.02038 (2024).

Dear Reviewer DwYd,

We provide a summary of our responses for your convenience and sincerely appreciate your insightful feedback.

1. Difference between token representation shrinkage and code collapse:

• Token representation shrinkage: the distribution of code assignments becomes narrower and fails to match the distribution of the original encoder embeddings.
• Code collapse: codes are rarely used or become inactive.
• Regardless of whether code collapse occurs, token representation shrinkage always reduces the representational diversity.

Our major contribution is the identification of previously unrecognized token representation shrinkage problem, along with a systematic analysis of its characteristics and impact.

2. Follow-up Experiments:

We conducted additional experiments according to your constructive suggestions:

• Added SimVQ [1] (ICCV 2025) as a new baseline.
• Include additional real-world datasets: ImageNet-1k, ODIR, and Oxford Flower 102.
• We report both Recall and Precision metrics for comprehensive evaluation.

In conclusion, our additional experiments (as shown in Rebuttal Table 1 - 12) further demonstrate that token representation shrinkage limits the representational diversity of codebook.

[1] Zhu, Yongxin, et al. "Addressing representation collapse in vector quantized models with one linear layer." arXiv preprint arXiv:2411.02038 (2024).

W1: My major concern lies in the misaligned evaluation metrics and datasets with common image generation works such as MaskGIT [4] and VAR [25].

W1.1: Evaluation metrics: Recall ("Improved precision and recall metric for assessing generative models". In NeurIPS 2019) is … [MASK]... It is recommended to report Recall numbers, which is widely used in existing literature. The other two missed evaluation metrics are IS and Precision (see [4] and [25]).

We appreciate the insightful feedback from the reviewer. We have included Recall, Precision and IS scores as measurement in the main text and attached as in Table 1, Table 2 and Table 3. VQ with reduced token representation shrinkage shows higher recall in generated outputs in VAR for ODIR and MaskGIT for ImageNet datasets.

Table. 1 VAR generative results on ODIR

Table. 3 MaskGIT generative results on Imagenet-100

Table. 1 VAR generative results on Imagenet-100

W1.2: Datasets: MaskGIT and VAR conduct experiments using ImageNet-1K with 256x256 and 512x512 resolution. It is acceptable to only conduct 256 resolution experiments if time is limited.

As shown in Table. 4 and Table. 5, we are currently conducting experiments on 256 × 256 resolution for Imagenet-1k. The preliminary results of the first few epoch are listed in table x and x. We come from an academic institution and only have access for 2-4 GPUs. Thus, we are still running the experiments can not obtain the final results in time.

Table. 4 LPIPS during training on Imagenet-1k

Table. 5 MSE during training on Imagenet-1k

We sincerely thank the reviewer for highlighting this important direction. We plan to extend our study to ImageNet-1K and potentially other real-world scenarios and will include this in the final version of the paper.

W2: A minor weakness is about the novelty of the proposed two-stage training solution. Recently, there exists some work (e.g. arxiv:2501.01423) leveraging existing pretrained vision models to constrain the latent space of the autoencoder during the autoencoder's training. Such methods, to some extent, weakened the novelty of the proposed "pretraining w/o VQ then fine-tuning w/ VQ" method in this paper.

We appreciate the reviewer’s observation and the pointer to related work. While similar two-stage strategies have emerged recently, we would like to emphasize that the core contribution of our work lies not in the pretraining procedure itself, but in the identification and analysis of the token representation shrinkage phenomenon. To the best of our knowledge, this issue has not been formally recognized in prior literature.

W3: Below are some comments on paper clarity (correct me if I'm wrong):

W3.1: The illustration of Figure 1 can be clearer. What are the meanings of the vertical axes of the histograms? What do the different colors stand for?

Thank you for pointing out the lack of clarity in the figure caption. We have revised it to be more informative in the updated manuscript.. In Figure 1, the vertical axis represents the density of token distribution, i.e., how often each token appears across different image classes.

The cyan bars indicate the ideal case where all classes are represented by a similar number of tokens, suggesting no token representation shrinkage. The red bars indicate the presence of token representation shrinkage, where some classes are overrepresented by tokens while others receive disproportionately fewer tokens. We will add additional illustration to clearer explain the phenomena.

The relationship between encoder representations (embeddings) and tokens in the codebook is not clear (Line 142-150 and Fig. 3). With untrained encoders, why are the embedding distributions in the bottom two figures of Fig. 3 (a) still well-expanded? And why are the token distributions (which are trained to be close to encoder embeddings) far from the embedding distributions in Fig. 3(a), unlike Fig. 3(b)?

The plots in Fig. 3 actually show the final states after training the VQ-VAE model. The only difference between Fig. 3(a) and Fig. 3(b) is how the codebook was initialized:

• In Fig. 3(a), the codebook was initialized from an untrained encoder.
• In Fig. 3(b), the codebook was initialized from a trained encoder using K-means over its output.

We observe that the encoder learns to produce semantically meaningful and well-dispersed embeddings after training, regardless of initialization. However, when the codebook is initialized from an untrained encoder, the initial codes are densely clustered. During training, the encoder updates rapidly, but the codebook updates lag behind and get trapped, leading to a situation where only a small subset of codebook entries tracks the encoder. As a result, the token becomes concentrated(token representation shrinkage occurs).

The introduction of the common initialization strategy seems to be ambiguous (Line 116). To me, initialization stands for the weights initialization of the codebook before training, which is a uniform initialization in common codebases, not the "K-means" initialization. Only after reading the context about the proposed method could I guess that the "initialization" here may indicate the codebook status at the beginning of the VQ codebook training process.

We thank the reviewer for pointing out the ambiguity. We have renamed “codebook initialization” to “setting initial values of codebook embedding vectors” in the updated text to avoid possible confusion.

Line 134: Fig.3 does not contain (c) (e).

We thank the reviewer for pointing out the typo and have revised the figure reference to Fig. 3(a) accordingly in the updated manuscript.

References

Reviewer 2

Q1:

We sincerely thank the reviewer for this insightful observation and the pointer to the Similarity Distribution Calibration (SDC) method by Ng et al. (BMVC 2023). Indeed, both token representation shrinkage in vector quantization and similarity collapse in hashing stem from over-concentration of representations in a limited space, leading to degraded semantic separability.

SDC addresses this issue by aligning the pairwise similarity distribution of binary codes with a predefined reference distribution (e.g., Beta distribution), thereby reshaping the similarity structure globally rather than preserving individual pairwise distances. This approach aligns well with our motivation to encourage token diversity and prevent overuse of a small subset of tokens.

While our method currently mitigates shrinkage via pretrained strategies, distribution-level regularization techniques like SDC offer a promising complementary direction. Specifically, incorporating pairwise similarity distribution regularization into the VQ training process could potentially enhance token dispersion and further stabilize training.

We will add a discussion paragraph in the final version of the paper to highlight this conceptual alignment and outline it as an exciting direction for future exploration.

Q2: do you also resize CIFAR10 dataset to 128x128? or perform with 32x32? if 32x32, what is latent size?

The image size is 32 * 32. And the latent size is 8 * 8.

Q3: what happens if the pretrained encoder is undertrained before VQ is introduced? Understanding the sensitivity to pretraining quality could clarify how much training is "enough" before enabling VQ.

Table 1. Validation loss(MSE ×10^-3) for AE.

Table 2. Performance of VQVAE initialized by different AE.

Thank you for the insightful question. We conducted a simple ablation to study the effect of pretraining quality. Our results show that even a short pretraining phase (e.g., 10 epochs) is sufficient to significantly improve the performance of VQ-VAE. Beyond 20 epochs, we observe that continued training of the autoencoder yields only marginal gains in VQ-VAE performance.

This suggests that the benefits of pretraining plateau after a certain point, and moderate pretraining is already sufficient to mitigate token representation shrinkage.

Reviewer 1

Q1: Could the authors explain the difference between codebook collapse and token representation shrinkage? If they are the same or similar things, are there other baseline methods to deal with codebook collapse that the authors could potentially compare against?

First, the codebook collapse [1, 2, 3, 4] and token representation shrinkage are distinct problems. Codebook collapse refers to the phenomenon where a portion of the codebook entries become "dead" (they are never or rarely used during training). In contrast, token representation shrinkage, which we newly formally identified in our manuscript, refers to a situation where the learned token embeddings crowd into a small region of the latent space, thereby limiting representational diversity. Token representation shrinkage can occur even when all codebook entries remain active. In such cases, although the codebook appears active, the effective token usage is still limited, resulting in a loss of expressiveness. Identifying the token representation shrinkage phenomenon is the major contribution in our study as it informs tokenizer design and behavior analysis of generative models.

In addition to clarifying that token representation shrinkage and code collapse are distinct issues, we further provide comparison results with SimVQ [4].

Table 1. Ablation on CIFAR-10 across different codebook sizes. MSE (×10^-4)

On the ImageNet-100 dataset (Table 2), our method also demonstrates better reconstruction quality, highlighting the effectiveness of our strategy in relative large-scale scenarios.

Due to resource limitation and time limitation, the generative results on Imagenet-100 have not finished yet. The initial validation loss in the first few epochs are shown in Table. 3.

Table 2. VAR tokenizer performance on Imagenet-100.

Table 3. Training Loss(Cross Entropy) for VAR

Besides CIFAR-10 and Imagenet-100, we also conduct experiments on ODIR and Oxford Flower 102. The results as shown in Table 4 to Table 7. We notice that our simple solution improves the diversity of generative model on different dataset. It is important to emphasize that the major contribution of this paper is the discovery of the phenomenon of token representation and understand how it affect the diversity of generative models’ outputs. We use a simplest method of pretraining encoder to show the effects. But the encoder pretraining method is not our major contribution. Therefore, it won’t be surprising to see advanced method such as SimVQ show better enhancement of diversity.

Table 4. VAR tokenizer performance on ODIR.

Table 5. VAR generation results on ODIR

Table 6. VAR tokenizer performance on Oxford Flowers 102. Cos Dis represents Cosine distance in codebook. Euc Dis represents Euclidean distance of codebook.

Table 7. VAR tokenizer performance on Oxford Flowers 102

Q2: Given the claim that "relatively small dataset size and limited inherent diversity" masks the impact of shrinkage, can the authors provide empirical support for this hypothesis or quantify dataset size/diversity in any way to better understand the impact?

Table 8. Mean of cosine distance for visual features on Imagenet-100 and ODIR

Table 9. Shrinkage ratio for ODIR and Imagenet-100

ODIR is ****a medical dataset containing only retinal images, which are naturally more homogeneous in content. To quantify dataset diversity, we computed the mean cosine distance between Inception-V3 features of test samples from both datasets. As shown in Table 8, ODIR exhibits significantly lower intra-dataset feature distances than ImageNet-100, indicating much lower visual diversity.

Furthermore, to assess the impact of token representation shrinkage ratio on output diversity, we computed the ratio of mean pixel distance between generated images with and without shrinkage As shown in Table 9, the shrinkage ratio in pixel-level diversity due to shrinkage is substantially more pronounced on ImageNet-100 than on ODIR.

Q3: Can the authors discuss why r-FID goes up with the VAR model on the ODIR dataset when shrinkage is not present?

We thank the reviewer for this thoughtful question. We offer the following possible explanations for the observed r-FID goes up. First, the ODIR dataset is relatively small(only 1200 images for test dataset), which may introduce fluctuations in FID measurements. Notably, as shown in the Table. 4 other metrics such as MSE and LPIPS, consistently improve after applying our pretraining strategy, indicating better reconstruction quality. Another reason is that the FID metric relies on features extracted from an Inception-V3 network pretrained on ImageNet-1K, which may not capture semantic differences accurately in domain-specific medical images such as fundus photos. This may affect the reliability of both FID and r-FID in this setting.

Q4: If the initialization strategy has an effect on token representation shrinkage, why did the authors not try different initialization methods to see if the problem can be alleviated?

We thank the reviewer for the valuable suggestion. We would like to clarify that the primary contribution of this work is the identification and analysis of the token representation shrinkage phenomenon, which has not been formally studied before.

To support this finding, we propose a simple yet effective pretraining-based initialization strategy that not only demonstrates the existence of shrinkage but also partially mitigates it. Our intention is not to exhaustively benchmark all possible initialization schemes, but rather to provide an initial and interpretable solution that highlights the significance of the problem.

We believe that exploring alternative initialization strategies is a meaningful next step, and we plan to investigate and compare them in future work.

References

[1] Yu, Jiahui, et al. "Vector-quantized image modeling with improved vqgan." arXiv preprint arXiv:2110.04627 (2021).

[2] Zheng, Chuanxia, and Andrea Vedaldi. "Online clustered codebook." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Huh, Minyoung, et al. "Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks." International Conference on Machine Learning. PMLR, 2023.

[4] Zhu, Yongxin, et al. "Addressing representation collapse in vector quantized models with one linear layer." arXiv preprint arXiv:2411.02038 (2024).

Dear Reviewer Zc2K

Thank you for your thoughtful and constructive feedback. We truly appreciate the time and effort you dedicated to reviewing our work.

You kindly suggested that addressing a particular concern might help improve the paper's evaluation. We have revised the manuscript accordingly and would be grateful if you could let us know whether our response resolves the issue.

If you have any remaining questions or concerns, we are pleased to provide further clarification. Thank you again for your time and valuable feedback.