Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning

Q1. Information loss from image rendering and trade-off between performance and computation cost:

1.Information Preservation in Rendering: The rendering process preserves all the words in the text image, ensuring that no textual information is lost during this step.

2.Information Encoding: Evaluating potential information loss during the embedding step is challenging as it operates at the feature level.

3. Performance vs. Computational Cost Trade-off: We acknowledge the trade-off between performance and computational efficiency. Using the original lengthy 426 tokens can outperform the rendered image for inference, but at a higher computational cost. To further investigate this trade-off, we conducted additional pre-training experiments using the smaller Mistral-7B model. We show the result below and find that with the same in-context text length and batch size, our method shows a slight decrease in performance (e.g., from 25.4 to 24.6 on average). However, our approach allows for longer in-context lengths and larger batch sizes with the same computational resources. When utilizing these advantages, the mean accuracy of our method clearly outperforms the baseline.

Caption: We compare the zero-shot evaluation of the OpenFlamingo-9B baseline model with our VisInContext. In this context, BSZ stands for batch size, and the prefix text token length is 128. With the same GPU memory, VisInContext supports a larger batch size or a longer in-context learning (ICL) length.

Due to time limit of rebuttal phase, we only conducted one analysis and will include more detailed discussion on this trade-off in the revised version of our paper.

Q2. Optimal compression rate: Our current implementation is already multiple text image inputs for longer texts by splitting the text into several images. This allows us to process texts longer than what a single image can accommodate. In Figure 2, we depicted only one image for simplicity and clarity.

The changes in font size and font interval directly correspond to the number of text images required. For example, using a smaller font size results in fewer images needed to render the same amount of text. This relationship indicates that adjusting font size and interval settings effectively controls the number of images used.

Q3. Comparison with other long context methods:

Thank you for insight comment.

[1]: This method involves an additional fine-tuning stage and modifies the cross-attention mask of the transformer. [2]: This method reuses the language model multiple times to obtain a summary vector for each segment and the training pipeline is very different. [3]: This approach requires an additional instruction fine-tuning stage on specific instruction data to produce the desired output.

These methods focus on re-use the pre-trained model by fine-tune on specific learning objective[1,2] / instruction data[3]. While these methods are not directly compatible with our pre-train (also increase ICL in this stage) and few-shot evaluation pipeline, these works have inspired us significantly. We are particularly interested in adopting the strategy from [2] during the inference stage for further increase the ICL. However, the code is hard to modify under our codebase and we are still working on it.

Regarding the pros and cons of other long context methods, we will expand our discussion in the revised manuscript to include: [4] and [5]: We will discuss these techniques in lines 287-289. For [6] and [7], we will cover these approaches in lines 290-292, which use positional interpolation to extend the context window.

Q4: Do we need text tokens? At the current stage, our method still relies on partial text input to perform next-word prediction and uses autoregressive language modeling loss to optimize the model. In this way, we can test downstream tasks directly by predicting next word. Completely replacing text tokens with images would require developing a unified learning target beyond the current contrastive loss. This is a complex challenge and an area of ongoing research.

Q5: If current method suit for larger models such as Llama-3:

The current approach is indeed suitable for larger models like Llama-3 405B, which has an in-context length of 8192 during pre-training (enabled by techniques like pipeline parallelism on 16 H100 GPUs and low bit quantization). These techniques are applicable to both vision and language models, allowing vision tokens to scale similarly.

Furthermore, the parameters of the vision model are only a fraction of the overall model parameters, making it feasible to include more visual tokens without significant overhead. For instance, the Llama-3 405B vision-language model incorporates a ViT-H vision encoder with 630M parameters, demonstrating the compatibility of our method.

Q1. Questions related to line 75-76.

i. Clarification on Token Concatenation: Notice that <visualx> is used as a placeholder to indicate the position of an image within the token sequence. It has a token length of 1. For example, in a sequence of 256 tokens, the structure could be <visual1><text1><visual2><text2><visual3><text3>, where the visual placeholders are interleaved with text tokens. In this example, 253 tokens out of the 256 are text tokens. If the text exceeds the length limit, it will be truncated during preprocessing.

ii. Pre-defined Hyperparameter $m$: The hyperparameter $m$ is predefined and set to a specific value, which is 3 in this work. It indicates the number of images included in a sequence. If there are not enough images to meet this value, zero tensors are used as padding to ensure the sequence length remains consistent at 3.

iii. Explanation of Corresponding Text and Image Tokens: The interleaved dataset, such as MMC4, includes multiple images and their corresponding texts. The "corresponding text" refers to the text paired with a specific image. In our notation, $I_i$ represents the $i$-th raw image. In Figure 2, visual information from both raw images and rendered text images is integrated into the LLM using a cross-attention mechanism. In this mechanism, the text tokens act as queries, while the keys and values are the visual tokens derived from the images, after applying token masking. Notice that $I_i$ only does self-attention with $T_i$ according placeholder <Visual i>.

Q2. Questions about Line 81-82:

The $M$ in these lines represents the number of rendered text images used. It is different from $m$, which is used to denote the number of raw images. We will use a different letter in the revised version.

Q3. Do not understand Table 1:

1. The details of ICL: The evaluation pipeline for MLLMs typically consists of a pre-training stage followed by a fine-tuning or zero-shot evaluation stage on downstream tasks. The "In-context Text Length" mentioned in Table 1 refers to the pre-training stage (Lines 160-162). For downstream tasks, we ensure a fair comparison by maintaining the same number of shots and input settings across all models. This means that the settings for both the baseline model and our method are identical during downstream evaluation, allowing us to accurately assess the benefits of longer context pre-training. Our method naturally supports longer shots during inference (Figure 4); however, comparing results between significantly different shot numbers, such as 128 shots versus fewer shots, would be unfair as more shots leads to better result in general. Therefore, we do not include these comparisons in Table 1.

2. Seems to be a degradation in performance with 0 shots in some cases, okvqa and vizwiz drops a lot : Each dataset has its own biases and evaluates different capabilities of the models. It is common for results to be slightly below the baseline in some datasets, but the overall mean performance is significantly better than the baseline. Please refer to Q4 for additional information on RVzob as a supplementary resource regarding the instability associated with increased shots.

Q4. Comparison against only two models:

Flamingo and Fuyu are two strong and representative methods: Flamingo represents models with visual encoders, while Fuyu represents models with only linear embeddings. All MLLMs fall into one of these two categories.

Thank you for your prompt feedback and for raising the score! We are glad that our rebuttal brings more clarity and answers all your questions. We would love to resolve any remaining clarity problems, if you could elaborate more about “there might be a problem of clarity”.

Q1. The motivation of “text-only in-context few-shots experiment” is not clear:

The motivation behind the “text-only in-context few-shots experiment” aligns with the common practices in mainstream few-shot learning architectures like Flamingo, IDEFICS, and EMU2[1]. These models often include text-only prompts for zero-shot evaluation.

For example, in a typical zero-shot evaluation setting, the input to the model is structured as <text 1><text 2><text 0><image 0>, where "0-shot" actually involves using 2-shot text-only prompts (Lines 170-173). This design helps bootstrap the output format, enabling the model to produce answers that follow the style of the prompts. Follow this line, we test the longer prompt beyond 2-shot text data.

So the primary reasons for using text-only prompts in these experiments are:

1. Testing Prompt Understanding: This setup tests the ability of model to clearly understand the prompt and follow instructions.
2. Practicality: It provides a practical advantage when prompt images are difficult to obtain, ensuring the model can still perform well in the absence of visual input.

Motivated by these, we propose to evaluate text-only in-context few-shot and demonstrate that text rendered as images can be comparably effective as raw text.

Q2. The details about token masking in Lines 87-88:

There seems to be a misunderstanding. The vision feature input to the cross-attention model is a sum of raw image tokens and rendered text image tokens. As depicted in Figure 2.

In our implementation, the raw image tokens are masked with a pre-defined probability, which is 50% in our case (details are provided in Lines 107-110). This means the model only observes the raw image tokens half of the time. The reference to a 1.0 masking ratio in lines 87-88 indicates that we mask all the raw image tokens when the model is not supposed to see them at all. However, the rendered text image tokens remain intact and still observed by the model.

The rationale behind this approach is that we experimentally found that models tend to overly rely on raw pixel images, often ignoring the information from rendered text images. By masking the raw image tokens, we encourage the model to pay attention to the rendered text images, thereby learning text-image semantics more effectively. For inference, we sum raw image token and rendered text image token (Line111-112).

Q3. Some other minor typos:

Thank you for pointing out these minor issues. We will fix the reference and figure compilation errors (Lines 159, 521) and polish the formats of Tables 1 and 2 to ensure they meet the required standards.

Q4. Explanation for Baseline Model Occasionally Performing Better:

Attention to Visual Tokens: For classification tasks, the model might prioritize distinguishing visual tokens over textual information. This could lead to better performance in scenarios where visual cues are more prominent.

More shots do not always result in better performance. As seen in related works like Flamingo (Table 1 in this work), it's not uncommon for more shots to occasionally result in worse performance. This variability can be due to the random selection of example shots from the support sets. If the selected examples are quite different from the query image-text pair, the model's performance might drop. Therefore, it is normal that the baseline outperforms our work occasionally and we mainly focus on mean accuracy over all datasets.

Some works, like RICES [2], analyze this phenomenon and focus on prompt ensembling by retrieving similar samples from the query set to highly improve multi-shot performance, which is not our focus.

[1]. Sun Q, Cui Y, Zhang X, et al. Generative multimodal models are in-context learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14398-14409.

[2]. Yang Z, Gan Z, Wang J, et al. An empirical study of GPT-3 for few-shot knowledge-based VQA. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(3): 3081-3089.

The masking ratio of 1.0 mentioned in Lines 87-88 differs from the predefined probability of 50% referenced in Lines 107-110.

The predefined probability of 50% in Lines 107-110 indicates that, during pre-training, there is a 50% chance that the raw image tokens will be masked.

The masking ratio of 1.0 in Lines 87-88 refers to the specific implementation of masking, where all raw image tokens are masked.

The purpose of Lines 87-88 is to address the issue where combining tokens from raw images and text images directly caused the network to overlook the text-image input (as mentioned in Lines 105-107).

Q1. Complexity of Implementation and Barrier to Adoption:

It is not true, and we would like to emphasize that our implementation is not only simple but also easy to adopt for future works.

i. Text Rendering: This step is performed during the preprocessing phase on the CPU and can be implemented in OpenCV with just a few lines of code. Specifically, our implementation requires only 5 lines of code.

ii. Token Masking: This is a straightforward selection process that can be implemented in a single line of code.

iii. Contrastive Loss: The computation of the contrastive loss is performed on the average token representations, which is a standard and easily implemented technique.

Overall, our method is designed to be simple and straightforward, ensuring it is accessible to practitioners without adding undue complexity.

Q2. Derivation from the initial purpose of LLMs; Need for such a complex paradigm; Engineering exercise:

We disagree with your assessment. Here are the key points:

1.Clarification on Purpose: It is not clear what you mean by the "initial purpose" of LLMs. Our focus is on Multi-modality Large Language Models (MLLMs), not solely LLMs. Our VisInContext method significantly increases the in-context text length for MLLMs during both the pre-training and inference stages.

2. Clarification on why we need such complex paradigm: Increasing the in-context text length is a crucial challenge, particularly during pre-training, while most works on LLMs require fine-tuning or post-tuning to support longer contexts. There is no prior work addressing this topic in MLLMs. It is widely recognized that vision encoders are significantly smaller than LLMs. Therefore, processing longer but not highly related text with a visual encoder at a very modest computational cost is an efficient approach. Actually, compared to rendered text images, text tokenizers in LLM require numerous human-defined preprocessing steps such as lowercasing, punctuation removal, stop words removal, tokenization, and more—generally involving almost ten steps. In this context, rendered text images offer a much simpler paradigm for processing text.

3. Clarification on engineering exercise: We use a fixed font and a simple white background. While font size does affect performance, we have already discussed this in our experiments. Our method significantly increases the in-context text length for MLLMs, representing a substantial improvement. This is not merely an engineering exercise but a strategic enhancement to the model's ability to process long text. Additionally, it is unclear what you mean by "engineering exercise".

In conclusion, our method addresses a critical need in MLLMs and provides tangible benefits, demonstrating its value beyond a simple engineering exercise. For MOE based MLLM, we increase the in-context text length from 256 to 2048 during pre-training stage.

Q3. Potential Bottlenecks of Using a Vision Encoder and Discussion of Limitations:

We argue that both the LLM and vision encoder have limitations in terms of processing long text sequence, especially for the purpose of extended context for multimodal understanding. Below we summarize the pros and cons of using vision encoder to process long text sequence.

1. Size Comparison: The vision encoder in an MLLM is typically much smaller compared to the LLM itself. For example, the vision encoder is 340M but the LLM is 56B for MOE-based MLLM. This size difference suggests that the vision encoder is a more economical way to process text information.

2. Efficiency: Rendered text images can encode more text within the same number of tokens (Lines 98-99), making this method efficient for handling longer contexts.

3. Complementary Technique: Our method is complementary to existing techniques for increasing in-context text length in LLMs. It provides an additional means to enhance model performance without replacing current methods.

4. Limitations and Trade-offs: We acknowledge potential limitations, such as dynamic token handling, which are discussed in Lines 302-304. However, our experimental results show that the benefits of our approach outweigh this limitation.

Q4. Concerns about redundancy in image space and the necessity of the "Text -> Text-Rendering" Process:

We disagree with the notion that projecting text into image space is suboptimal due to redundancy.

1.Redundancy in Real-World Images vs. Rendered Text Images: While real-world images often contain redundancy because adjacent patches look similar, include colors, textures, background noise, which can be redundant and not directly related to the underlying semantic meaning. But for rendered text image, since the image is primarily composed of meaningful text with minimal background details, it closely mirrors the semantic density of the text itself and significantly reduces the typical redundancy found in image space.

In addition, process text with image have following advantages:

1. Efficiency of Image Tokens: In our method, the number of image tokens is fewer than the equivalent number of text tokens for the same length of input (Line 98-99). This demonstrates that processing text in image space can be more efficient than using text tokens alone.

2. Economic Efficiency: Our approach shows that preprocessing text into image space is more economically friendly compared to relying solely on text models. For example, flops in Figure 1.