UniTok: a Unified Tokenizer for Visual Generation and Understanding

Dear Reviewer emrz,

Thank you for your constructive feedback and thoughtful evaluation of our work. We hope our point-to-point response will address your concerns.

Q1: Fair comparison with RQ.

We apologize for any misconceptions caused. In Table 7, we do implement RQ with the multi-code setting (i.e., using 8 codes to represent a single token), consistent with the MCQ approach. For RQ, the codes are generated by recursively quantizing each token, so the number of codes is determined by the quantization depth (D) rather than the number of codebooks (K). Following the official RQ implementation, we set D = 8 and K = 1 for the experiments in Table 7. This results in 16 × 16 × 8 codes per image, matching the configuration used in MCQ. We will clarify this point in the revised version of our paper.

Q2: Comparison with VAVAE.

Thank you for the suggestion. We will include VAVAE for comparison in Table 2. However, we would like to clarify that VAVAE should not be considered a baseline for UniTok, as it is a continuous VAE tokenizer evaluated within the diffusion framework, whereas UniTok is a discrete VQVAE tokenizer evaluated under the autoregressive framework. This fundamental difference makes the two approaches less directly comparable. Instead, LlamaGen is a widely adopted baseline for discrete tokenizers. In Table 2, we strictly follow the LlamaGen implementation and only substitute the tokenizer with UniTok for evaluation. In this context, we believe that using LlamaGen as a baseline provides a more fair and meaningful comparison.

Q3: Why choose Liquid rather than VILA-U as the MLLM framework for evaluation?

• First, we would like to clarify that the earliest release of Liquid was on December 5, 2024, on arXiv, which is more than five months before the NeurIPS 2025 submission deadline.
• Besides, the training data of VILA-U contains 15M internal data that are not publicly available. This makes it impossible to conduct controlled experiments with VILA-U.
• We attempted to request access to the internal data used in both Liquid and VILA-U. However, only the authors of Liquid granted us access. Therefore, we selected Liquid as our evaluation codebase.

Q4: Does the performance in Table 3 come from better training data?

We agree that data quality has a significant impact on final performance. However, most unified models today use different data recipes for training and often involve internal data (e.g., as in Chameleon, VILA-U, Show-o, Janus, Liquid, among others). This makes direct, apple-to-apple comparisons impractical. Nonetheless, we believe the tokenizer innovation in UniTok is the primary contributor to the final performance, as demonstrated in Figure 3.

Q5: Evaluating UniTok on the GenEval benchmark.

Thank you for the suggestion. The table below presents a comparison of unified models on the GenEval benchmark. The overall trend is largely consistent with the results observed on GenAI-Bench, except that Show-o achieves better performance by scaling up the text-to-image training data to 2 billion samples. Notably, UniTok achieves non-trivial improvements over Liquid while using exactly the same set of text-to-image training data.

#Data: Number of text-to-image data samples used in pretraining

Thank you for your feedback and for taking the time to review our response. We are glad that our reply addressed some of your concerns. If there are any remaining issues or questions, please let us know—we would be happy to provide further clarification.

Dear Reviewer uzhd,

Thank you for your constructive feedback and thoughtful evaluation of our work. We hope our point-to-point response will address your concerns.

Q1: Clarification on the vocabulary size in Table 8

• First, we would like to clarify that the row of Codebook in Table 8 denotes the codebook size rather than the vocabulary size. (We apologize for any confusion caused by the notation.) In UniTok, the vocabulary size is exponentially larger than the codebook size. For example, UniTok uses 8 codebooks with 4096 code entries each, resulting in a total codebook size of 4096 × 8 = 32,768. In contrast, the vocabulary size is 4096^8 = 2^96, i.e., there are up to 4096^8 possible combinations of codes for each token.

• We have extended Table 8 below to illustrate how both the vocabulary size and latent code dimension increase with the number of codebooks. As shown, increasing the number of codebooks simultaneously scales up the vocabulary size and latent dimension, which consistently leads to improved performance. We will revise Table 8 in our manuscript to make this relationship clearer.

Codebook	1×16384	2×8192	4×4096	8×2048
Vocabulary	2^14	2^26	2^48	2^88
Latent Dim.	8	16	32	64
rFID ↓	1.50	0.98	0.54	0.33
Accuracy	41.0%	43.9%	44.7%	46.1%

Q2: Inconsistency between observations in L144 and the conclusion

Although joint training can initially lead to a degradation in classification accuracy and reconstruction FID (L144), we observe that this degradation diminishes after improving the quantization methods (L147). For example, Table 6 shows that joint training actually results in better rFID and gFID. This supports our conclusion that VQVAE training and CLIP training do not inherently conflict; rather, the underlying issue is that discrete tokens lack sufficient capacity to simultaneously encode both low-level details and high-level semantics.

Q3: The impact of increasing the latent code dimension (single codebook setting)

• Thank you for your suggestion. The table below presents a comparison between the multi-codebook and single codebook settings, where the latent code dimension is scaled up to 64d. Due to time constraints, we trained the tokenizers on OpenImages using only the reconstruction loss. As shown, solely increasing the latent dimension of a single codebook results in lower codebook utilization rate (even with entropy loss) and degraded rFID.

Codebook Size	Vocabulary Size	Latent Dim.	Codebook Utilization Rate	rFID ↓
1×32768	2^14	64	82%	2.20
8×4096	2^88	64	100%	0.33

• Besides, as mentioned in Lines 45-47, previous works have also reported similar effects when scaling the latent dimension. Specifically, Table 4 in ViT-VQGAN [1] provides a detailed and quantitative ablation study supporting this observation.

[1] Yu, Jiahui, et al. "Vector-quantized image modeling with improved vqgan." arXiv preprint arXiv:2110.04627 (2021).

Dear Reviewer rTfH, Thank you for your constructive feedback and thoughtful evaluation of our work. We hope that our responses below will address your questions and concerns.

Q1: How does contrastive loss improve generation?

In UniTok, the contrastive loss (CLIP supervision) injects semantic information into the code embeddings, which is conceptually similar to the MAE loss in MAETok [1] and the VF loss in VA-VAE [2]. Both of these works have demonstrated that semantic guidance leads to more structured latent distributions, which in turn facilitate diffusion-based generative modeling. Our experimental results in Tables 2 and 6 further confirm that this benefit extends to discrete latent spaces and autoregressive generative models.

[1] Yao, Jingfeng, Bin Yang, and Xinggang Wang. "Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[2] Chen, Hao, et al. "Masked autoencoders are effective tokenizers for diffusion models." Forty-second International Conference on Machine Learning. 2025.

Q2: How does reconstruction loss impact understanding?

Intuitively, reconstruction loss encourages the tokenizer to preserve low-level details (e.g., texture) when encoding images. Since the total representation capacity of discrete tokens is limited, this creates a trade-off — allocating more "storage space" to low-level details leaves less capacity for encoding high-level semantics, which can lead to degraded understanding performance. However, when the discrete latent space is enlarged (i.e., by using multi-codebook quantization), this effect becomes minor and has only a mild impact on understanding performance.

Q3: Related work ViTok

Thank you for the suggestion. ViTok presents a comprehensive study on the scaling properties of VAEs, offering valuable insights into VAE design. We appreciate your recommendation and will include a discussion of this work in our revised manuscript.

Dear Reviewer Qdyc,

We greatly appreciate your insightful comments and support for our work. We hope that our responses below will address your questions.

Q1: Resolution used in LMM evaluation (Table 6)

In Table 6, we evaluate the model at 256×256 resolution for all tasks, consistent with the resolution used during UniTok training. We have also tested UniTok at 384×384 and 512×512 resolutions within the LLaVA framework. We found that UniTok generalizes well to 384×384 resolution without finetuning, likely due to its hybrid architecture (CNN + ViT). In particular, accuracy on the TextVQA benchmark—which benefits from higher resolution input—increases by 2%. However, further increasing the input resolution to 512×512 yields little additional improvement. We believe that finetuning UniTok at higher resolutions is necessary to make the most of high-resolution inputs.

Q2: Could UniTok be enhanced by incorporating captioning loss?

That's an excellent point. In addition to the image-text contrastive loss, image captioning loss is generally considered beneficial and is widely adopted in CLIP training, as demonstrated by works such as CoCa, BLIP2, and SigLIP2. Since UniTok inherits the CLIP training paradigm, we believe that incorporating captioning loss could further enhance UniTok's performance. Besides, TexTok [1] has shown that image captions can complement visual tokens to achieve better reconstruction quality and higher compression rates. Therefore, leveraging generated captions during the image decoding process could potentially yield additional improvements.

[1] Zha, Kaiwen, et al. "Language-guided image tokenization for generation." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.