A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation

Dear Reviewer 3Gvw,

Thanks for your detailed and valuable comments and suggestions. It is encouraging to see that you find our work well-founded and efficient. We are here to address all your concerns:

W1: The paper’s theoretical analysis is significantly lacking.

In our design and experiments, DnD-Transformer is used to predict the multi-depth VQ codes produced by RQ-VAE[1], which are proved to significantly improve the reconstruction ability of VQ visual tokenizers. In this work, we do not propose a new visual tokenizer or improve the original RQVAE. DnD-Transformer is proposed to more effectively predict the multi-depth codes during generation. In the Original RQVAE paper[1], they use two separate transformers to predict the codes, as one transformer predicts the first layer, the other transformer predicts the rest. For DnD-Transformer, as shown in Figure-5, the multi-layer codes for within the same sequence index are predicted in one forward pass. We hope our response can address your concerns and please feel free to raise specific points about the insufficient explaination.

W2: The experimental setup for the arxiv-images presented by the author raises some confusion.

We set two tasks for rich-text image generation, the PIL-images and arXiv-images. The arXiv-images is far more difficult than the PIL-images task as it involves complex font size, style, color and graphical items, and DnD-Transformer shows non-trival results (it can output valid sentences under this conditon) while the diffusion baselines all fail to do it. We think this is a promising signal showing the AR image generation and DnD-Transformer are more suitable in the rich-text generation task. In the PIL-image experiments, DnD-Transformer shows signicantly better results than Diffusion model, even if the SD3 has better visual tokenizers that can reconstruct rich-text image better according to table-1b. The two experiments jointly show that DnD-Transformer is more suitable in the task of rich-text generation tasks compared to the diffusion baselines.

W3: Why didn’t the author employ models like SiT or DiT to validate DnD’s effectiveness?

We choose SD3 since it was the sota diffusion model, we want to directly compare with the best to show the advantage of DnD-Transformer on rich-text image generation tasks. Following your advice, we also conduct experiments with DiT by training it from scratch to generate PIL-Image and arxiv-Image in a unconditional manner. The new results are shown in section H in the appendix. And the conclusions are the same that they lag far behind the DnD-Transformer in rich-text generation tasks.

W4: I did not find sufficient evidence in the experiments to convincingly show a reduction in VQ loss.

In our paper, Figure-3 and Table-1 show that with more depth of the RQ-Tokenizer, the VQ loss (L2 loss) becomes smaller and result in better image reconstruction performance both in natural and rich-text images.

W-others:

Thanks for your mention, we added the results of MAR[1] to the main table and add to the related work. The reconsturction result are shown in Figure-3 and Table-1 to validate that VQ with more depth would lead to better image reconstruction.

We hope our response resolves your concerns and to hear your feedback. If you have further questions, we are always here to respond. Thank you again.

Thanks for your reply.

W1 and W4’s responses still leave some questions unanswered. While I understand that the authors’ work builds on the follow-up research of RQ, they have not provided a clear rationale for why a single transformer might outperform RQ’s two independent prediction modules. Admittedly, this issue does not diminish the overall value of the paper.

Regarding W4, my original question was whether the proposed method improves reconstruction compared to the baselines, such as RQ or VQ. The authors have clarified in their response to W1 that their approach does not aim to enhance Visual Tokenizers. However, I noticed that Section 2.2 repeatedly references the DnD tokenizer. Could the authors elaborate further on its role, e.g., can I expect a better Table 1 with baseline RQ?

As for W2, I find the response somewhat insufficient. I was hoping for examples demonstrating that images generated using DnD genuinely achieve Vision-Language Intelligence, as suggested by the title. Specifically, I expected DnD to enable the text within the generated rich-text images to be coherent and meaningful. However, Fig. 7 does not align with these expectations. Could the authors provide further evidence to substantiate their claim in Section 4.4 of “A Spark of Vision-Language Intelligence”?

I thank the author for providing additional results for W3.

Dear Reviewer BF3N,

Thanks for your detailed and valuable comments and suggestions. It is encouraging to see that you find our work intriguing and potentially offering a more principled and human-like learning approach. We are here to address all your questions, clarify some misunderstandings and fix the overstated statements.

Weakness 1: "Advantage over with RQ-VAE"

DnD-Transformer does not improve RQ-VAE as a visual tokenizer. Instead, it uses the RQ-VAE as the tokenizer. Its biggest advantage over the RQ-VAE paper is that it is a single end-to-end transformer that predicts the multicodes without expanding the sequence length, unlike the RQ-Transformer, which uses two individual transformers to predict the codes. This simple yet effective structure is more efficient in both training and inference, and it can be better integrated with current LLM structures without involving additional models. The novelty is also acknowledged by reviewers Zh5x, FQZc, 3Gvw, and 4s2g.

Weakness 2: "I believe the manuscript requires extensive revision"

Thanks for your advice. We have changed the citation to official publications and explain the difference between DnD-Transformer and RQ, HQ-Transformer. We will further explain these in our response to question #3 and #4.

Weakness 3: "I believe many experiments in this paper may lead to misinterpretation"

The comparison is fair since the VAE of SD3 actually has better text reconstruction ability than our trained RQ-VAE, according to Table 3. The rOCR of the SD3 VAE is better than our best 8-layer RQ-VAE trained on rich-text images. However, even with a better visual tokenizer, diffusion models still lag behind our DnD-Transformer in the rich-text generation task. This evidence better justifies our claim that DnD-Transformer outperforms LDM on this task. We use the same VQ training pipeline as RQ-VAE without changing the database. We will further explain these points in our response to questions #5 and #8.

Question 1: Wording problem

We think there might be some misunderstanding about our wording. We use the word "resurgence" to imply that AR image generation methods like DALL-E1, Muse, and RQ-Transformer were popular, as you mentioned, "before ChatGPT," but were later replaced by diffusion models such as DALL-E2 and Stable Diffusion. However, recently (after ChatGPT), there has been a tremendous amount of work, such as Chameleon[1] , Emu3[2], which use more advanced LLM structures such as LLaMA to conduct AR image generation tasks. This is why we call it a "resurgence".

Question2: "The claim in the phrase “a spark of vision-language intelligence for the first time” appears to be overstated"

Thanks for your mention. We have removed the claim from the paper and replaced with more accurate "it shows that we can conduct accurate language modeling with pure image generation model."

Question3: "How is the ICR of JPEG computed?"

ICR of JPEG was calculated by the ratio between number of bits used to store the compressed image and the original image. As JPEG has an additional parameter to control the compress rate, (1-100, higher means less information loss), we use the common compression rate reported by [3] and report the ICR of 5%~10%, which is significantly higher than that of VQ. We have fixed it in the paper.

Dear Reviewer 3Gvw,

Thanks for your detailed and valuable comments and suggestions. It is encouraging to see that you find our work interesting and with strong directive idea. We are here to address all your concerns:

W1 & Q1: objective data on FLOPs, training time, and inference speed

Thanks for your mention, we have added the detailed benchmark on training/inference complexity compared to different baselines (Table-5). All experiments are conducted on A100 GPUs.

Actually, DnD-Transformer only adds additional prediction head (linear projection) to the backbone transformer model. Assume the dimension of hidden state of the LLM is H, the vocab size is V, and there are N additional output heads. DnD-Transformer only adds HxVxN parameters, which is only a very small portion of parameters comparing to the LLM backbone according the above table. During training and inference, only N additional linear projection and softmax operation are conducted.

W2 & Q2: Only explores class-to-image and unconditional image generation.

Our main baseline LlamaGen[1], utilizes 50M Laion-coco + 10M internal data to conduct text-2-image generation experiments. We are unable to get internal data and train on such a large dataset due to limited computation resources. We think the ImageNet and Rich-Text image generation can already show the effectiveness of our method compared to the LlamaGen baseline.

Q3: generalize to higher resolutions without exploding complexity?

The answer is yes. As mentioned in W1&Q1, compared to the baseline AR generation LlamaGen, DnD-Transformer only adds HxVxN parameters to the backbone model and only N additional linear projection and softmax operation are conducted, which are irrlevent of the image input resolution.

[1] Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

[2] Scalable Diffusion Models with Transformers

[3] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

We hope our response resolves your concerns and to hear your feedback. If you have further questions, we are always here to respond. Thank you again.

Dear Reviewer FQZc,

Thanks for your detailed and valuable comments and suggestions. It is encouraging to see that you find our work interesting and acknowledge our achievements. We are here to address all your concerns:

Weakness 1: Some implementation details are missing

If we understand you right, you are mentioning the details in Question 3. Please refer to Q3 for our response and feel free to raise other questions about implementation details.

Weakness 2: Some baselines are missed

Thanks for your advice. We have added VAR to the main table and related work.

Weakness 3: About more vision-language ability

We have observed some interesting phenomena in our experimental rich-text generation experiments, as shown in Figure 20 in the appendix. The DnD-Transformer, when trained to generate pure text images, exhibits a generation pattern similar to LLMs. It can understand the names of celebrities such as Barack Obama and knows he is related to the United States, but it also tends to repeat and hallucinate. We agree that it is worth exploring more vision-language tasks in the future.

Question 1 : About depth vs performance

We conduct the gFID vs. depth experiments on PIL-Text and Imagenet datasets. The results are shown below.

We conduct experiments on the same Transformer-XXL structure with the distance of each code-output layer to 3.

With increased depth, the PIL-Text generation result could be further improved. For imagenet, as we discussed about it in section 4.5 in the original paper that the increase in depth does not always bring in better generation quality in our experiments. The result on ImageNet256 shows that as the depth of code increases, the reconstruction performance keeps improving, however the relative improvement become marginal as the depth goes deeper. The generation performance first improves (from 1 to 3) then degrades (from 3 to 4). We think the result is due to the marginal improvement in rFID when the depth is large. Within the same backbone model, the overall prediction complexity and error accumulation becomes larger , but the theoritical performance improvement is marginal (rFID) when the depth grows.

Dear Reviewer Zh5x,

Thanks for your detailed and valuable comments and suggestions. It is encouraging to see that you find our work creative and have significant advantage over other methods. We are here to address your concerns with our updated experiments results and clarification:

Weakness1: Evaluation of the DnD-Transformer in terms of sensitivity to layer indexes

We have incorporated the results of DnD-Transformer-XXL using various output layer indices, while maintaining a 2-layer configuration. The final code output layer is consistently aligned with the last transformer layer, while we vary the layer index for the initial code output layer. The table below illustrates the relationship between the distance of these two code output layers and the resulting FID score for generation on ImageNet256

The conclusion drawn from our study indicates that the DnD-Transformer is relatively robust across a broad spectrum of layer indices, ranging from 3 to 12, with only minor variations in gFID scores (between 2.53 and 2.64). However, when the gap between the layers is too narrow or too wide, the results tend to diminish. The XXL model comprises 48 layers in total. Our findings suggest that more layers should be allocated for predicting the first layer of code, indicating that the first-layer code is more challenging to predict than the later-layer code. This conclusion aligns with our analysis of code distribution, as shown in Figure 4(a, b). The image demonstrates that the first layer code utilizes a broader range of codes (Figure a shows the first layer employs 100% of the entire codebook, whereas the second layer uses approximately 70%) and displays a higher variance in code selection (Figure b). This broader usage and higher variance suggest that the first-layer code's distribution is more complex than that of the second layer, thereby increasing the prediction difficulty. Consequently, more layers are needed to predict the first code than the second code.

Additionally, when we explored the configuration where the number of heads equals the number of layers, we observed that the generation quality considerably deteriorated under this setup (with gFID scores of 23.55 for #layers=1 and 26.79 for #layers=2). We believe the cause is linked to the necessity of having more layers to accurately predict the first code, which in turn enhances the overall generation quality. In comparison to the aforementioned results, when 46 additional layers are allocated for predicting the first-layer code, the gFID improves significantly from 26.79 to 5.12.