Thank you for your detailed review and valuable comments. Please note our top-level comment. Below we address specific questions.

Qualitative results

Please see our top-level comment. We present more qualitative results in the attached pdf file. Note that image reconstruction capabilities and image generation capabilities may not be aligned. Our primary focus is on image generation.

Generalizability

The primary focus of this paper lies in the AutoRegressive (AR) IG framework (as mentioned by line 27 in the main text). We are revising our manuscript to further clarify the scope of our work.

We also conduct preliminary experiments on Diffusion Models and observe that IU capabilities may enhance their performance. We initially follow LDM [9] to train a VQGAN and a VQ-KD CLIP model. Both models adopted 8-dimensional codebooks of size 16384. Subsequently, we trained diffusion models for 20 epochs on IN-1k for conditional image generation. We summarize the performance of VQGAN, VQ-KD CLIP, and their corresponding diffusion models below:

We find that the FID score of the VQ-KD model (at 31.31) is marginally lower than the FID score of the VQGAN model (at 31.67). Yet, it is important to note that the current VQ-KD and Cluster methods are tailored for token-based generators, because token-based frameworks are more apt for exploring the interplay between Image Understanding (IU) and Image Generation (IG). Consequently, these methods may not be directly applicable to continuous VAE models. We remain on the lookout for alternative frameworks that could effectively incorporate IU to assist continuous VAE models.

Comparison Experiments

Currently, image tokenization methods can be roughly separated into two groups: vector-quantization-based and scalar-quantization-based. VQGAN adopts vector quantization, while FSQ is one of the latest image tokenizers that adopt scalar-quantization. Other works like SQ-VAE and CVQ-VAE introduce various improvements to VQGAN, but share the same architecture as VQGAN, so their impact on our final conclusion is little.

For comparison with related works, we adopt a stronger recipe to train the AR proposal network, with larger batch size (32 per GPU), more training epochs, and classifier-free-guidance (CFG). As shown in the following table, VQ-KD CLIP outperforms prior image tokenizers:

Presentation

Our work studies the relationship between IU and IG. We adopt this organization to clearly express our findings. Any better suggestions are welcome.

Larger Teacher Models

In Tab. 6 of the main text, as the OpenCLIP teacher gets larger, the FID AR metric consistently decreases. The FID AR metric evaluates the similarity between generated images and reference images (usually the IN1k val split), so we use FID AR as the primary metric. Other metrics including rFID, PPL, and IS AR, evaluate the reconstruction quality, quality of token sequence modeling, and diversity of the generated images. These metrics may be subject to noise disturbance, so a larger teacher does not guarantee better metrics. Nonetheless, larger teachers do tend to yield better metrics.

Codebook Sizes and Dimensions

Larger codebook sizes or dimensions do not necessarily lead to stronger IU models. For instance, in Tab. 7 (b) of the main text, increasing the codebook dimension leads to a significant drop in codebook usage, which limits the IU capabilities of VQ-KD.

To further investigate the relationship between IU and IG capabilities, we conduct linear probing experiments on IN1k, where tokenizers with higher Top-5 Acc. metrics tend to achieve better FID AR metrics:

The above results suggest that tokenizers with stronger IU capabilities tend to behave better in IG tasks.

I have carefully read your rebuttals and the rebuttals have solved my questions. The new visualizations can reflect that IU model helps IG tasks well. The experiments on diffusion models can also demonstrate this paper's conclusion(though diffusion is not actually what this paper discusses). The comparisons and linear probing experiments about IU capability are clear. And it is good for you to do such detailed experiments during this time. I think this paper is worth accepted.

Thank you for your constructive comments. Please note our top-level comment. Below we address specific questions.

Theoretical Support

While it is hard to theoretically prove the superiority of the VQ-KD tokenizer, we hypothesize that superiority is because each token generated by VQ-KD contains rich semantics, rather than simply appearance information. The hypothesis is supported by the finding that the PPL AR metric of most VQ-KD tokenizers are lower than VQGAN and FSQ in Tab. 1 of the main text. Lower PPL AR metric suggests that the proposal network can easily model the token sequence generated by VQ-KD. Moreover, Fig. 4 in the main text demonstrates that images belonging to the same category are encoded with similar tokens. Please also refer to Fig. 2 in the attached PDF file for a qualitative analysis of the semantics encoded within each VQ-KD token.

Generalizability

Please see our top-level comment. We perform experiments on challenging datasets including LAION-Aesthetics. Visualizations are shown in the attached PDF file. The results demonstrate that our conclusion generalizes well to various datasets.

Computation cost

We present the computation cost for each training stage of VQGAN and VQ-KD in the following tables. All experiments are conducted on IN-1k using 8 A100-80G GPUs. In sum, VQGAN takes 81 hours to train and VQ-KD CLIP takes 89 hours to train. The training cost for VQ-KD CLIP is only 10% higher than VQGAN.

Thank you for your detailed review and thoughtful comments. Please note our top-level comment. Below we address specific questions.

Related works

Our main conclusion is that IU models can aid IG tasks, which have not been explored before. While LaViT adopts a pretrained ViT encoder in the tokenizer, they did not perform in-depth analysis regarding the relationship between IU and IG. We will modify our claim and clarify that we are the first to demonstrate that IU models can substantially enhance IG through VQ-KD.

Main observations

We perform an ablation study on the quantization operation of VQ-KD and show the results in our top-level comment. The results suggest that VQ-KD can optimize the quantizer in the same way as VQGAN, while achieving better IG performance.

As for codebook usage, Tab. 7 (b) in the main text shows that when the codebook dimension of VQ-KD increases to 256, which is the same as VQGAN, the codebook usage of VQ-KD drops to 48.7%. However, VQ-KD still achieves 12.08 FID AR, which is significantly better than the 24.11 FID AR metric of VQGAN.

To support the conclusion that Tokenizers with stronger semantic understanding tend to deliver superior IG performance, we further conduct linear probing experiments on IN1k:

As shown in the above table, VQ-KD CLIP and VQ-KD ViT achieve the highest Top-5 Acc. scores, while VQ-KD MAE achieves the worst Top-5 Acc metric. This trend in the Top-5 Acc. metric is roughly the same as the trend in the FID AR metric, suggesting that tokenizers with stronger IU capabilities tend to behave better in IG tasks.

Qualitative results

Please see our top-level comment. We present more visualizations and quantitative analysis for the VQ-KD codebook in the attached PDF file.

Presentation

Thanks for the suggestions. We are revising our manuscript and will use Tab./Fig. for cross-reference. In Eq. (4), the symbol $\mathcal{D}$ represents the decoder of VQ-KD, which maps the code vectors $\mathbf{C}(z)$ to the feature space of teacher $\mathcal{T}'$. We apologize for the confusion and will provide additional information in the main text.

Comparison Experiments

We adopt a stronger recipe to train the AR proposal network, with larger batch size (32 per GPU), more training epochs, and classifier-free-guidance (CFG). As shown in the following table, VQ-KD CLIP outperforms prior AR and NAR methods, and is comparable to some Diffusion-based methods.

Dear Reviewer,

Thank you for the suggestions that help us improve the paper. As the deadline for discussion is approaching, please let us know if you have any additional questions. We genuinely hope you can consider raising the score if we have satisfactorily addressed your concerns.

Thanks again,

The Authors

Thank you for your thoughtful feedback. Please note our top-level comment. Below we address specific questions.

Generalizability

Please see our top-level comment. We examine VQ-KD ConvNext and observe good image generation abilities. More types of IU encoders will be added to our revised manuscript. We also conduct VQGAN and VQ-KD CLIP experiments on medical imaging (SA-Med2D-20M) and satellite imagery (SATIN) datasets, where VQ-KD CLIP consistently outperforms VQGAN. Visualizations are shown in the attached PDF file. These results confirm the generalization ability of our conclusions across different models and datasets.

Ablation Study

Please refer to our top-level comment, where we replace the K-Means module in VQ-KD with the quantization loss used in VQGAN. A slight performance drop is observed, but the overall performance of VQ-KD still surpasses VQGAN by a large margin.

Fidelity-Related Metrics

Some results in Tab. 7 of the main text shows that the generation ability of VQ-KD can surpass VQGAN even if its rFID metric is relatively lower. To further support this observation, we conduct experiments with a VQ-KD CLIP model that is not well-trained. As shown in the following table, while the rFID metric of this VQ-KD CLIP model is worse than VQGAN, its PPL and FID metrics still outperform VQGAN by a large margin.

Qualitative results

Please see our top-level comment. We present more qualitative results in the attached pdf file.

VQGAN description

Thanks for pointing out this ambiguity. We will clarify this in our revised manuscript to avoid confusion. VQGAN adopts perceptual loss to enhance the perceptual quality of reconstructed images. Given a real image $\mathbf{I}$ and a reconstructed image $\hat{\mathbf{I}}$, the perceptual loss encodes both images and outputs the distance between the image features. While semantic constraint is introduced, the primary goal of perceptual loss is to achieve better pixel reconstruction.

Dear Reviewer,

Thank you for the suggestions that help us improve the paper. As the deadline for discussion is approaching, please let us know if you have any additional questions. We genuinely hope you can consider raising the score if we have satisfactorily addressed your concerns.

Thanks again,

The Authors