W1: More details on two-stage training?

Please see "General Questions and Concerns - Details of two-stage training." At the warm-up stage, the model's input is still RGB images, and output is the proxy codes, with cross-entropy loss for supervision.

W2: Ablation test of 2D variant of TiTok-B64?

We thank the reviewer for the question. Both the TiTok-B-64 (2D) and TiTok-B-64 (1D) use the same number (64) of tokens, except that 2D variant kept the patch tokens as latent grid representation and directly reconstructed images from these 2D tokens (similar to a common 2D tokenizer), whereas 1D variant uses the latent tokens to guide the reconstruction from the mask token sequences. This experiment demonstrates that 1D formulation shows superior performance to 2D counterparts, especially when the number of tokens is limited.

Q1: How important is the off-the-shelf codebook? Continuous autoencoder instead of VQGAN as the outside compressor?

Please see "General Questions and Concerns - Reviewer HiP7 Q1: How important is the off-the-shelf codebook? Can we use a continuous autoencoder?"

W1&2: two-stage training method, Improved public available tokenizer training recipe?

Please see "General Questions and Concerns - Reviewer Mn4Y W1&2: Two-stage training and better single-stage recipe from Open-MAGVIT2?"

Q1: Comparison to SEED?

We thank the reviewer for the suggestion. SEED is already cited and discussed in the Related Work L102 to L109. Specifically, BLIP, SEED, EMU, and other similar multi-modal LLMs build a tokenizer on top of CLIP encoder, which gives highly-semantic tokens. However, due to the nature of CLIP models that focus on high-level information, they can only feed the tokens into a diffusion model and only reconstruct an image with high-level semantic similarities, while the layouts and details may not be well reconstructed. (see Fig. 3 in the SEED paper as a reference).

W1: Why proxy code (two-stage training) instead of VQGAN loss (recon + LPIPS + GAN)?

It is noteworthy that the two-stage training is not necessary, as is demonstrated in "General Questions and Concerns-Single-stage training for TiTok". We also show that TiTok can work well with the commonly used Taming-VQGAN recipe as shown in Tab.3c. However, as is discussed in L345 - 352, there is a performance gap between Taming-VQGAN recipe and MaskGIT-VQGAN recipe, where the latter one has a significantly better performance but provides no public reference or access. We adopt the two-stage training mainly for bridging the gap between the Taming-VQGAN recipe and other state-of-the-art tokenizers.

W2: Ablation studies on hyperparameters for sampling

Following prior arts [10, 67], we did a grid search (with step 0.5) on the sampling hyper-parameters and reported the optimal ones. We add a further ablation study based on TiTok-L-32 as suggested below (the final setting is labeled in bold, each grid number is organized as IS/FID). The effects of using CFG or not is already provided in Tab. 6 in the submission ("w/o guidance" refers to no CFG and "w/ guidance" refers to CFG).

W3: Uncurated visualizations

Thanks for the valuable comments, we provided uncurated visualizations in the attached PDF file.

Q1: Why is gFID better than rFID?

We appreciate the good question. This is because rFID is computed against the real ImageNet val set, i.e., we compute the FID score between the reconstructed val set and the real ImageNet val set. However, gFID is computed against the virtual ImageNet reference statistics from OpenAI's ADM [D] (see "Sec 2.2 Sample Quality Metrics" in their paper). This is also why the gFID of the real val set is 1.78 instead of 0. The rFID and gFID are commonly correlated but they are not directly comparable. It is also noticeable that the evaluation protocols we follow are widely used by most prior works [4, 10, 35, 48, 54, 67].

Q2: Why not use auto-regressive transformer for generation?

As discussed in the limitation section, we use MaskGIT generation framework as it is much more efficient compared to diffusion models or auto-regressive models, e.g., DiT-XL/2 is a 675M model, ViT-VQGAN is a 1.7B model, and the recent VAR requires training a 2.0B model for SOTA performance, while our model using MaskGIT only trains a 177M model and achieves better or comparable performance to aforementioned counterparts.

Q3: "Token number mismatch" in the proxy code training?

Please see "General Questions and Concerns - Reviewer 8XiP Q3: How is TiTok trained with Proxy codes?"

[D] Dhariwal, et.al. "Diffusion models beat gans on image synthesis." NeurIPS 2021

W1.1: Grounding experiment and analysis of 1D tokens, advantages of 1D tokens against 2D tokens. What if we mask a subset?

Main advantages of 1D tokens are "semantic-meaningful" and "more compact". The "semantic-meaningful" is grounded by experiments in Fig. 4c, FID score in Tab. 1, 2, and visualization in Fig. 7, where we show that TiTok tends to learn more semantic-meaningful representation with a limited number of tokens. For "more compact", as evidenced by Fig.4d and Tab. 1, 2, TiTok uses much fewer tokens and leads to a substantial generation speed-up.

Masking a subset could be challenging. As there is no concept of "pad token" or "placehold" in the learned codebook (regardless of 1D or 2D), it is not doable to "mask a subset" and then reconstruct an image. The best we can do is to randomly replace a subset of tokens with random tokens, yet it is still challenging to provide meaningful insights or comparison between 1D and 2D tokenizers.

W1.2: Reducing sizes with 2D tokens

We emphasize that the scaling experiments aim at studying the model's behavior under different numbers of tokens, which is not necessarily related to 1D or 2D tokens. However, 1D token representation provides a flexible design to use arbitrary number of tokens, while 2D tokenizers often limit the token number choices among $k^2$ (where $k^2$ is one of [1024, 256, 64]), making it not suitable to study the scaling property with different number latent tokens. Besides, as shown in Tab. 3c, where the 2D variant of TiTok-B using 64 tokens ($8^2$) achieves significantly worse performance than the 1D variant of TiTok-B using the same number of tokens, indicating that 2D tokenizers are not good choices with limited number of tokens.

W2: More evaluation metrics for reconstruction

FID and IS are the main metrics to evaluate the reconstruction in the context of tokenizer and generator [10, 35, 54, 64]. As suggested, we report additional metrics obtained using the same code-base to ensure fairness.

The slightly worse PSNR/SSIM scores of TiTok variants match our observation that although TiTok can retain most important/salient information of the image, the high-frequency/details may be ignored/made up, as compensation for using fewer tokens. However, an improved training recipe (see “General Questions and Concerns - Single-stage training for TiTok”) can significantly boost all these metrics.

W3: More details on decoder-finetuning:

See “General Questions and Concerns - Reviewer Pxnc W3: Explanation of decoder fine-tuning?”

Q1: Potential advantages and disadvantages of 1D v.s. 2D

1D tokenizer allows more flexible design and more semantic-meaningful and compact tokens compared to 2D, significantly speeding up the generation process while maintaining competitive scores. It is noteworthy that one may use the same large number of tokens in 1D tokenizer, thus making it a direct alternative to existing 2D tokenizers with similar or better performance.

The main disadvantages are 1D tokenizers are far more under-explored, compared to 2D tokenizers, demanding for more research efforts to study its applications to different tasks (e.g., image editing, diffusion models, video generation, multi-modal LLM, etc.). Besides, the training paradigm of 1D tokenizer could be further refined, e.g., while we adopt the two-stage training in the paper, we currently observe promising results from an improved single-stage training recipe as suggested by Reviewer Mn4Y, which could be further improved.

Q2: Other generation models (e.g., diffusion)?

As discussed in the limitation section, we use MaskGIT generation framework as it is much more efficient compared to diffusion models or auto-regressive models, e.g., DiT-XL/2 is a 675M model, ViT-VQGAN is a 1.7B model, and the recent VAR [D] requires a 2.0B model for attaining SOTA performance, while our model with MaskGIT only requires a 177M model and achieves better or comparable performance to aforementioned counterparts.

The TiTok framework is totally compatible with the KL-VAE formulation that is widely used by diffusion models. As this paper mainly focuses on the tokenizer part, we believe the combination with diffusion models can be a promising future direction.

Q3: Comparison to Vision transformers need registers?

Thanks for the suggestion. We will cite the paper (denoted as ViTreg below) for a discussion in a revision. TiTok uses latent tokens as the image representation, similar to ViTreg and other prior works. (a detailed discussion is available in L102-109). However, the two works have significant differences. ViTreg uses a set of latent tokens to allow a cleaner and more interpretable attention map, as the added latent tokens can help alleviate the artifacts/outliers in the original self-attention map. On the other hand, TiTok explicitly uses latent tokens to encode all the information needed to reconstruct the original image, where the latent tokens can be regarded as an information bottleneck between the tokenizer and de-tokenizer. The two works have distinctly different motivations and focuses.

W4: Writing improvement

Thanks for the valuable suggestions and we will revise accordingly.

[D] Tian, Keyu, et al. "Visual autoregressive modeling: Scalable image generation via next-scale prediction." arXiv:2404.02905 (2024).

W1.1: 2D v.s. 1D in image generation

We appreciate the valuable insights and comments. We note that the "2D advantages" of "higher resolution images and better effects" mainly stem from more tokens being used. While the 1D formulation in this paper mainly aims at a compact tokenization, we note that when using the similar number (perhaps slightly fewer) tokens, it is expected to work on par, if not better, to 2D counterparts.

W1.2: 1D tokenization for multi-modal large language model

We appreciate the suggestion and we strongly agree that TiTok has great potential in the multi-modal large language model. However, due to limitations on computation resources (e.g., The VQ-tokenizer-based multi-modal large language model Chameleon [C] by Meta requires training on 1024 A100-80G for 1 month), we thus mainly verify the tokenizer's effectiveness on the generation tasks, following prior arts [19, 35, 64, 66, 67]. We leave the application to multi-modal models as promising future directions.

W2.1: Measure the information encoded by tokens

We thank the reviewer for the valuable questions. We tried our best to provide quantitative and qualitative measurements on the information carried by 1D tokens. For example, in Fig.4.c, we show the linear probing accuracy (and thus correlates with the "semantic" in the tokens) when the number of tokens varies. We also provide visualization on different numbers of tokens in Fig. 7 where we observe that the tokens tend to carry important/salient regions when the number of tokens is limited. Although we agree that it will be intriguing if we may measure each token individually, it may require further research efforts and we leave it for future work.

W2.2: What specific information is forgotten? 2D relationship, translation, rotation? High-level dictionary?

It is noteworthy that our work is a very simple and initial attempt at this promising direction. As mentioned in W2.1, it is very challenging to directly "visualize" what role each token is responsible for, or how 2D relationship is modeled in the tokenization, although they are interesting topics towards better explainability. Based on experiments from Fig.4 and visualization from Fig. 7, we observe that the model tends to forget the low-level, high-frequency details in the background. For a similar reason, though the tokenizer can handle certain levels of translation/rotation in the input images, it is tricky to figure out which tokens are responsible for this part. High-level dictionary may not be capable to faithfully reconstruct the original images while TiTok tokens aim at a better trade-off between compact and semantic tokenization and faithful reconstruction (Please also see our response to Reviewer mn4Y’s Q1 for detailed examples on reconstructing images from CLIP high-level features, which fail to reconstruct the image layout or details).

W3: More figures?

We appreciate the suggestions. We provided more visualization in the attached PDF file. As promised in the paper, we will also open-source the code & model for the community to examine.

[C] Team, Chameleon. "Chameleon: Mixed-modal early-fusion foundation models." arXiv preprint arXiv:2405.09818 (2024).