Making LLaMA SEE and Draw with SEED Tokenizer

W1-1: Write loss functions.

Response: Thank you for the advice. We have revised the method section and included all loss functions in our revised manuscript.

W1-2: Why use two different image encoders.

Response: We use two different image encoders in order to ease training by leveraging the pre-trained knowledge of BLIP-2 and unCLIP-SD. Specifically, we initialize the Causal Q-Former from the pre-trained BLIP-2 Q-Former that is compatible with BLIP-2 ViT, so we use the ViT encoder from BLIP-2 for image encoding. And we would like to decode images using the off-the-shelf SD (frozen), so we adopt the image encoder (disposable after training) from unCLIP-SD to provide the learning target for our generation embeddings. After training, the generation embeddings can be used in place of the original conditional embedding of SD-UNet.

Since the image encoder of unCLIP-SD is frozen during training, only the parameters need to be loaded onto GPU, occupying 2944MiB memory (12.7% of total memory 23260MiB for batch size 512).

It is theoretically feasible to use only the unCLIP-SD encoder - Causal Q-Former requires random initialization. We will leave it for future work.

W1-3: The importance of contrastive loss.

Response: It is critical since the Causal Q-Former is trained using only the contrastive loss. Specifically, the training of visual tokenizer consists of two stages: 1) Causal Q-Former, and 2) VQ Codebook and MLP.

We have attempted to employ a single-stage approach to optimize all components but the codebook was hard to be optimized and eventually collapsed. This may be due to the continuous evolution of the reconstruction targets of discrete codes (i.e., the features of the Causal Q-Former).

To conclude, for stable training, the Causal Q-Former needs to be optimized individually whereas contrastive learning acts as the sole objective and thus cannot be removed.

W1-4: The training objectives of VQ.

Response: Sorry for missing details of training VQ codebook. We adopt (1) the original VQ codebook loss, (2) a reconstruction loss between reconstructed causal embeddings and causal embeddings of Causal Q-Former, (3) a reconstruction loss between generation embedding and image embedding of unCLIP-SD to optimize VQ codebook. To avoid codebook collapse, in addition to the two-stage training method mentioned above, we follow BEiT v2[1] to reduce the dimension of codebook embeddings to 32D. We use the stop-gradient operation in the original VQ codebook loss. We have revised the method section with detailed introduction.

[1] BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

W2: The performance of bilateral visual codes when the number of tokens is enforced to 32.

Response: Such an operation is applicable only when the model predicts equal to or more than 32 tokens. We have attempted to truncate the visual tokens in such cases, however, there are still 2 failure cases given 5000 captions in COCO test set. The results of two models when the number of image tokens is enforced to be always 32 are listed as below. We can observe that the model with causal visual codes achieve higher similarity between generated images and groundtruth images (i2i), and between generated images and captions (i2t) than the model with bilateral visual codes.

W3: How to process videos.

Response: We train the SEED tokenizer with only image-text pairs (i.e., the Causal Q-Former and generation embeddings are only trained on image data), and only use video-text data for training SEED-LLaMA.

When we train SEED-LLaMA on videos, we sample four frames from each video. Each frame is discretized into 32 codes. The input sequence of a video is <IMG> the token of frame 1 </IMG> <IMG> the token of frame 2 </IMG> <IMG> the token of frame 3 </IMG> <IMG> the token of frame 4 </IMG>.

Q1: Why 32 tokens and 8192 codes?

Response: The numbers are selected based on the practice in existing works, i.e., 32 queries in BLIP-2[1] and a visual vocabulary of 8192 codes in BEIT V2[2].

[1] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models [2] BEIT V2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

Q2: Start and end token for images and videos.

Response: We use a start and end token for each image as shown in Fig.4. The same start and end token is used for both images and video frames (mentioned in the response to W3). We have included this information in section 3.2.1.

Q3: Suggestions.

Response: Thank you a lot for the useful suggestions. We have mentioned the codebook size in section 3.1.2, and added corresponding description and reference in the revised manuscript.

W1: The information leakage by bidirectional attention in ViT.

Response: The bidirectional attention in ViT is among 2D patches while the unidirectional causal attention in Causal Q-Former is among 1D queries. It signifies that the information of the current query is derived based on the context provided by previous queries. It is worth noting that the causal queries capture sparse-to-dense semantics from the entire image rather than only a part of an image, as demonstrated in Fig. 9 of Appendix. So it is ok for each query to see all ViT features via cross-attention. Also, according to the information flow (image$\rightarrow$frozen ViT, w/ init queries$\rightarrow$Q-Former), the semantics of $k$-th query would not be leaked to $\{k-1\}$-th query via ViT.

W2: SEED-LLaMA-I (14B) does not surpass SEED-LLaMA-I (8B) on some Image-Text tasks.

Response: These traditional VQA benchmarks (VQAv2, OKVQA, VizWiz) fail to properly reflect the model's multimodal capabilities since they require an exact match between the model prediction and the ground truth, as pointed out by recent works[2,3,4]. However, empowered by LLM, the model predictions that have been penalized may even be better than the ground truth with a single word or phrase.

We can observe that SEED-LLaMA-I (14B) outperforms SEED-LLaMA-I (8B) on SEED-Bench, which is specifically designed for evaluating MLLMs with open-form output. Such results have indicated the scalability of SEED-LLaMA.

The same phenomenon is also reported in IDEFICS (TODO CITE), where IDEFICS-I (80B) achieves worse performance than IDEFICS-I (9B) on VQAv2, OKVQA, VizWiz but much better on SEED-Bench. Check out the results below.

[1] Obelics: An open web-scale filtered dataset of interleaved image-text documents [2] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension [3] MMBench: Is Your Multi-modal Model an All-around Player? [4] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

W3: Need FID for generation evaluation.

Response: We would like to clarify that FID is mostly adopted to measure the quality of generated images by comparing the distribution between a set of generated images and a set of real images. However, to properly evaluate our SEED tokenizer, we focus more on the semantical consistency between the original input image and the reconstructed image. FID fails to meet the requirements, and CLIP similarity is more suitable in such a case, which is also adopted by GILL[1]. Furthermore, SEED aims to better tokenize images and uses the off-the-shelf SD-UNet decoder. Quality is generally determined by the decoder and is not specifically optimized in our work. We further evaluate the text-to-image generation on MS-COCO with FID as the evluation metric as below. We can observe that our model achieves competitve performance.

[1] Generating Images with Multimodal Language Models

W4: "Causal Visual Codes vs. Bilateral Visual Codes" in understanding tasks.

Response: We evaluate the performance of zero-shot visual question answering tasks on VQAv2, OKVQA and GQA (accuracy as the metric), and image captioning on COCO (CIDEr as the metric) as below,

We can observe that the model with bilateral visual codes significantly drops the performance in understanding tasks. When only considering understanding tasks (text as the only output) for training, bilateral visual codes have a better theoretical performance due to the rich semantics. However, in SEED-LLaMA, where the understanding (text as output) and generation (image as output) tasks are unified, the autoregressive Transformer fails to be properly optimized when using bilateral codes as it struggles with next-visual-token prediction for generation tasks. As a result, it achieves inferior performance on understanding tasks as well.

W1: Not religiously explained and self-contained. Sec. 3.1 is hard to follow. No equation.

Response: Thank you for the suggestion. We designed the SEED tokenizer with two premises: (1) image tokens should capture high-level semantics, and (2) image tokens should be produced under 1D causal dependence. The architecture of SEED meets the above requirements with the sub-modules of a pre-trained CLIP-ViT encoder, a Causal Q-former, a VQ codebook, and a De-Tokenizer which is composed of an MLP and the pre-trained SD-UNet.

Intuitively, the pre-trained CLIP-ViT encoder captures high-level semantics, the causal Q-former produces 1D causal embeddings, which are subsequently discretized into 1D causal IDs using the VQ codebook. The produced visual IDs enable LLM to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction. The de-tokenizer is used to reconstruct images, employed in case the follow-up multimodal LLM (i.e., SEED-LLaMA) predicts image token IDs and needs to render them into pixels further. Note that, to pursue efficient training, we directly adopt the pre-trained and frozen SD-UNet decoder, and train a lightweight MLP to transform the visual codes to the conditional embedding space of UNet.

During SEED training, only Causal Q-Former, VQ Codebook, and the MLP are tunable. There are two training stages. First, we optimize the causal Q-Former by image-text contrastive loss in the form of InfoNCE. Then, we jointly train the VQ codebook and the MLP. Besides the original VQ objectives, we adopt dual reconstruction, i.e., the reconstruction of continuous causal embeddings (to avoid code collapse) and the reconstruction of conditional generation embeddings (to be compatible with the off-the-shelf SD-UNet). The auxiliary text encoder and image encoder (disposable after training) are derived from BLIP-2 and unCLIP-SD respectively, providing learning targets.

We have carefully revised Sec. 3.1 in blue, adding equations and objective functions for each learnable component (Enq.1 for Causal Q-former, Enq.2 for VQ codebook, and Enq.3 for Multi-layer perceptron). Please find more details in our revised manuscript.

W2: Lack of analysis of Table 1 and Table 3.

Response: Thank you for the suggestion.

(1) Regarding Table 1: We would like to first clarify that our goal is not to surpass BLIP-2, since it is a comprehension-only model whereas our SEED is tailored for unifying comprehension and generation in one framework. It is worth noting that to achieve the goal of unification, we would inevitably sacrifice the performance of comprehension to achieve autoregressive multimodal generation. Specifically, the 1D causal dependence (unidirectional attention) and discrete tokens are both designed to accommodate generation but are harmful to discriminative image representation learning. Here we took BLIP-2 as a baseline for comparison since we adopted its pre-trained ViT and Qformer as initialization (mentioned in Sec. 3). As expected, our causal embeddings (before VQ) achieve better results in terms of image-to-text retrieval due to more training data, while falling short in text-to-image retrieval that relies on more discriminative image features. Generally speaking, with our elaborate training, the Causal Qformer achieves competitive results even with unidirectional attention.

(2) Regarding Table 3: The table benchmarks the ability of multimodal comprehension. Although we achieve multimodal generation at the expense of discriminative representations (discussed above), we still obtain competitive comprehension results compared to SOTAs and much better performance than the ones that also unify comprehension and generation (such as CM3Leon and Emu). SEED-LLaMA generally complies with the scaling law, i.e., more parameters lead to better results as shown in SEED-Bench (a benchmark tailored for MLLMs). And we found that the traditional benchmarks (e.g., VQAv2) are no longer sufficient to properly evaluate the MLLMs due to even better predictions than the ground-truth.

More explanations can be found in our revised manuscript.