Image Generation with Channel-wise Quantization

Thank you for your careful review and helpful suggestions!

Q1: However, channel-wise quantization is widely used in various applications, such as classification [a], LLM compression [b], and super-resolution [c].

A1: I disagree with this statement. The motivation and methodolgy are different. The channel-wise quantization used in [a], [b] and [c] belong to model quatization, which is a technique to accelerate model inference. While, the channel-wise quantization for image generation aims to learn visual representation.

Q2: Why not combine SD series for comparison? For example, SD1.5, SDXL and SD2.1.

A2: As shown in section 4.3, we train our model only with MS-COCO (83K) dataset following U-ViT. However, SD series are trained with large-scale datasets. It is unfair to compare with SD series directly, due to the large data gap. Besides, to our knowledge, the FID of SDv2.1 on COCO is 12.43, refer to [1]. It lags behind our model (5.85).

Q3&Q6: More experimental results about high-resolution image generation should be provided to demonstrate the effectiveness of the proposed method. The authors only conducted experiments on MS-COCO and ImageNet datasets with the image resolution 256*256.

A3&A6: As shown in Table 3, we provide the class-conditional image generation on ImageNet 512x512, not only the image resolution 256x256. The results on ImageNet 512x512 demonstrate the effectiveness of the proposed method. For higher image generation, like 1024x1024, it spends much time to implement, since we need to train channel-wise tokenzier for 1024x1024. Besides, there is an option to implement image generation on 1024x1024. We can use a super resolution model to upscale image from 512x512 to 1024x1024. We leave it for future work.

Q4: The authors did not evaluate the text-image alignment between the text prompts and the synthesized images.

A4: We are conducting the evaluation on text-image alignment. We will update results as soon as possible.

Q5: It is not reasonable to develop a channel-wise tokenizer, which is limited to one specific image resolution. One general channel-wise tokenizer algorithm that could be suitable for all image resolutions is favored. Such limitation heavily limits the contribution of this work.

A5: I disagree with this statement. First, for our method, it is an alternative to scale codebook. For codebook scaling, we conduct an experiment with codebook size 2^17=131072 without entropy regularization. The model obtains 1.33 rFID and 100% usage on ImageNet. This result validates the advantage of our method on codebook scaling. Second, although a model is trained for one specific image resolution, we can train different models for any image resolution. Our method is not limited to one specific image resolution, as shown in Table 2 and Table 3.

Q7: Marginal performance.

A7: We update our results in the paper. With widely used sampling scheme, our models achieve higher performance than VAR on FID and IS scores, as shown in Table 1 and Table 2.

Q8: The best results under the same setting in Table 2 should be bold.

A8: Thanks for your suggestion. We have updated it in our paper.

Q9: Figure 2 is a little blur and seems that some parts are screenshots of other images.

A9: Thanks for your suggestion. We will update it in the final version.

Q10: What is the purpose of the image generation? Only evaluating the image quality of the synthesized images based on evaluation metrics like FID, SSIM and PSNR is not enough. Can the synthesized images promote the downstream visual perception performance such as classification?

A10: I disagree with this statement. In image generation field, almost all works evaluate the image quality of the synthesized images based on evaluation metrics like FID, SSIM and PSNR. Thus, it is enough to validate the effectiveness of our method using these metrics. Using the synthesized images to promote the downstream visual perception performance is an interesting future work.

[1] Dong, Runpei, et al. "Dreamllm: Synergistic multimodal comprehension and creation." arXiv preprint arXiv:2309.11499 (2023).

Q4: The authors did not evaluate the text-image alignment between the text prompts and the synthesized images.

A4: we use CLIP score to show text-image alignment performance. The results are as follows:

MS-COCO 256x256

To our knowledge, the CLIP score of our method shows a good text-image alignment performance.

Thank you for your careful review and helpful suggestions!

Q1: The motivation of proposing channel-wise quantization.

A1: As shown in section 3.2 Discussion, spatial tokenizers pay more attention to local image areas and are easily to collapse into a limited number of visual units. If image tokens concentrate on larger image areas, image tokens can be more diverse, resulting in high usage for codebook. For channel-wise tokenizer, it forces image tokens to focus on large image areas, due to its large receptive filed. Thus, channel-wise tokenizer is an alternative to codebook scaling.

Q2: The paper lacks a detailed analysis of how the channel-wise tokenizer behaves differently from spatial tokenizers.

A2:

Image structures: As shown in “Comparisons with other image tokenizers”, our tokenizer demonstrates superior results on SSIM score than spatial tokenizer. The higher SSIM score shows that our channel-wise tokenizer captures image structure.

local details: As well known, IS score is used to evaluate the quality of generated image. Higher IS score means the more details of the generated images. As shown in Table 2 and Table 3, our models achieve higher scores on IS than all previous works. These results show that our channel-wise tokenizer captures local details.

Q3: Entropy regularization

A3: To validate our channel-wise quantization on codebook usage, we conduct a experiment using codebook size 2^17=131072 without entropy regularization. The model obtains 1.33 rFID and 100% usage on ImageNet. This result validates that channel-wise quantization is the main factor behind high codebook usage.

Q4: What does a channel token typically represent? Is it possible to visualize the learned channel tokens?

A4: We have add visualization of codebook tokens in Appendix A1. As shown in Figure 4, our channel- wise tokenizers can capture image structure and local details simultaneously. Please refer to A.1 for more details.

Thank you for your careful review and helpful suggestions!

Q1: Channel-wise quantization lacks theoretical justification and Intuitive rationality. Without spatial-based quantization, the model training appears to lose its causal nature

A1:

Theoretical justification and Intuitive rationality: As shown in section 3.2 Discussion, the spatial tokenizers pay more attention to local image areas and are easily to collapse into a limited number of visual units. If image tokens concentrate on larger image areas, image tokens can be more diverse, resulting in high usage for codebook. For channel-wise tokenizer, it forces image tokens to focus on large image areas, due to its large receptive filed. Thus, channel-wise tokenizer is an alternative to achieve high usage of codebook.

causal nature: I disagree with this statement. As demonstrated in MAR [1], masked generative models used by our method are generalized autoregressive (AR) models, having causal nature. As shown in experiments, our method shows high performance on image generation. It demonstrates our image tokens are compatible with these generalized AR models. From this point, our channel-wise tokenizer do not lose causal nature. For strict AR modeling, we leave it for future work.

Q2: As the model scales up, the number of tokens would need to increase, potentially making learning more difficult.

A2: We evaluate the scalability of our method from two perspectives. First, based on 16384 codebook size, we train a large model (1.0B) to validate the scalability of our method. Our model achieves 1.91 FID and 344.9 IS, outperforming VAR (2.0B) on these two metrics. Second, we conduct a experiment using codebook size 2^17=131072 without entropy regularization. The tokenizer obtains 1.33 rFID and 100% usage on ImageNet. This result validates the scalability of our channel-wise tokenizer on codebook size.

Q3: The resulting tokenizer becomes incompatible when image resolution changes

A3: Although a model is trained for one specific image resolution, we can train different models for any image resolution. Our method is not limited to one specific image resolution, as shown in Table 2 and Table 3.

[1] Li, Tianhong, et al. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

Thank you for your careful review and helpful suggestions!

Q1&Q2: Fair evaluation wrt compression ratios, more analysis on various token dimensions. The token count in Table 5 is potentially misleading, as the models use different compression ratios.

A1&A2: In table 5, for VAGAN and MaskGiT, they use 256x16x16 tokens (256x256). The usage of tokens is the same as Ours. The comparison is fair. For LlamaGen, it use 8x16x16 tokens. We use this type tokens for LlamaGen, due to its higher performance. If LlamaGen uses 256x16x16 tokens, the rFID is 9.21, which is worse than ours (1.64). We have updated the Table 5 to clarify this issue. For Table 2, Table 3 and Table 4, the compression ratios are 16, 32, 16, respectively. Thus, tokens used by our model are all 256 (16x16). The comparison is fair.

Q3: The ablation study for token dimensions is incomplete. Experiments within the range [8, 256] are needed to understand scaling behavior better.

A3: We add the result of token dim 32 into Table 6 (a). In this setting, LlamaGen achieves 3.22 rFID and 20.9% usage. This result is worse than one with dim 8. Combining with other results, LlamaGen can not perform better than one with dim 8 in range (8, 256].

Q4: Tokenizing global information along the channel dimension implies a significant correlation among tokens, meaning each image is learned as a whole rather than in parts. Consequently, C tokens per image are kind of memorized, making it difficult to repurpose tokens for other images due to global encoding. This also explains the high overall usage of the codebook, as tokens are not easily reusable.

A4: First, for our method, due to large receptive fields, image tokens are forced to capture large image areas and more diverse than ones generated by spatial tokenizers. This is the methodology of our channel-wise tokenizer. Second, I disagree with the difficult reusable for our image tokens. As shown in Table 6 (b), our tokenizer with 1024 codebook size can achieve 2.25 rFID. This means we can use 1024 tokens to represent 50000 images (ImageNet eval set) at least. Since an image uses 256 tokens, 50000 images means 12.8M image tokens. If our 1024 tokens are not reusable, how to represent 12.8M image tokens?

Q5: Adding codebook size, embedding dimensions, and compression ratios to the tables would improve comparability.

A5: Thanks for your suggestion. We have added some necessary information into Table 5 to clarify the fair comparison.

Q6: In Table 5, why is rFID significantly better while PSNR is worse, even though SSIM is better? I would expect a consistent trade-off between perceptual- and pixel-wise metrics.

A6: The reason is two folds. First, the codebook size is not large enough to represent all the details of images. When we use larger codebook size(65536), PSNR is increased to 18.72. Second, the tokens representing an image is also not large enough. As shown in Table 5, with 512 tokens, PSNR is increased to 21.47. Meanwhile, rFID and SSIM are also improved largely. The improvement on all metrics shows that there is not a consistent trade-off between perceptual- and pixel-wise metrics.

Q7: Exploring channel-wise quantization in autoregressive (AR) or diffusion tasks could be insightful. I assume it may not perform as well for AR tasks, as the AR function would need to predict the entire global image in one step rather than progressively building it up from parts.

A7: I disagree with the assumption. For our channel-wise tokenizer, it forces image tokens to capture large image areas but not the whole image. If image tokens generated by our tokenizer represent the whole image, they are not reusable in other images. As shown in A4, tokens in codebook are reusable, it challenges the assumption.

Q8: The claims regarding low similarity between channel tokens, efficient modeling of image structure, and strong representational capacity lack clear definition and verification.

A8:

low similarity between channel tokens

This is validated by our 100% codebook usage under different codebook size, especially 2^17 codebook size. If there is high similarity between channel tokens, codebook usage can not reach 100%.

efficient modeling of image structure

This is validated by high SSIM score, demonstrated in Table 5. SSIM is sensitive to image structure changes. High SSIM means efficient modeling of image structure. We have discussed it in “Comparisons with other image tokenizers”.

strong representational capacity

First, for image reconstruction, our tokenizer outperforms baseline (LlamaGen) on rFID and codebook usage, as shown in Figure 1, Table 5 and Table 6 (a). Second, for image generation, generators with our tokenizer outperform other generators, like VAR. These both demonstrate strong representational capacity of our tokenizer.

Q9: I have ignored going into details regarding quantitative results mainly because the issue of fair evaluation is not cleared yet. As of now, sometimes the model is better, sometimes worse, and it is unclear why and when one would want to choose this method over the spatial tokenization.

A9: We clarify the issue of fair evaluation in A1&A2. With the updated results, our method based on channel-wise tokenizer outperforms other generators, like VAR. Besides, our tokenizer can reach 100% codebook usage, especially 2^17 codebook size. Based on high generation performance and 100% codebook usage, our tokenizers is an alternative to spatial tokenization.