Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Thank you for recognizing the strengths of our approach and the efforts we put into enhancing OCR capabilities and data collection for multimodal language models.

For comparison with baselines/methodology novelty/Evaluation metric/Evaluation Scenarios, please refer to the general response.

Whether OCR-specific data is needed:

Compared to hundreds of millions of text-image pairs used in mPLUG-owl, we acknowledge that the scale of our data is relatively limited, which is also relatively affordable for most academic labs. We presume that training on large-scale non-OCR data improves OCR performance, as many of the captions in LAION datasets are equivalent to incomplete OCR results (Texts in an online image will sometimes appear in the captions). In the scale of our experiment, we observe similar improvement that just training on captions of text-rich images can help with text recognition capability (In Table 3, variant (4) is better than variant (1)). However, training on captions only (variant (4)) is not as good as training on OCR-based data (variant (2)(6)), at least in the scale of our experiments. We assume such a training signal can be strong enough for the scale of mPLUG-Owl. In general, we believe OCR-specific data is necessary for data efficiency.

Ablation study on the LLaVA benchmark

We find that including pretraining data improves the conversation capability, probably because longer training data leads to generating longer responses (Table 1). On the other hand, including finetuning data only hurts the conversation capability but increases complex reasoning. Combining pretraining and finetuning data improves the trade-off between conversation and complex reasoning. Generally speaking, GPT-4-based evaluation is not very robust, as there are some clear clues that it favors long responses [1]. By providing results on the LLaVA benchmark, we prove that incorporating our data will at least not harm the performance of interacting with natural images.

Why based on LAION rather than well-annotated OCR datasets:

The well-annotated OCR datasets are usually restricted to certain domains such as black-and-white documents, book covers, etc. As shown in Appendix Figure 10, the text-rich images in the LAION dataset, which contains all kinds of text-image pairs from the internet, are diverse and usually interleaved with natural images. We believe our instruction-following dataset based on LAION suffers from a relatively small domain shift compared to previously collected instruction-following data based on COCO, thus beneficial for potential knowledge transfer (Section 5.4). Also, we believe that collecting data based on real-world documents and well-annotated OCR datasets is an important next step to extend the scope of the data.

[1] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data, 2023.

Thanks for the responses of the authors, which have solved most of my concerns.

I would like to lift my score if the authors provide a revised paper with the new experiment results.

Best.

Thank you for acknowledging the organization and clarity of our paper, as well as appreciating the advancements we've made in data collection and our detailed analysis.

For comparison with baselines/methodology novelty/evaluation scenarios, please refer to the general responses.

LLAVAR achieves comparable performance with mPLUG-Owl, a model that was not trained on a text-rich dataset? :

It is hard to conclude that mPLUG-Owl is a model not trained on a text-rich dataset, as many of the captions in LAION datasets are equivalent to incomplete OCR results (Texts in an online image will sometimes appear in the captions). In the scale of our experiment, we observe similar improvement that just training on captions of text-rich images can help with text recognition capability (In Table 3, variant (4) is better than variant (1)). However, training on captions only (variant (4)) is not as good as training on OCR-based data (variant (2)(6)), at least in the scale of our experiments. We assume training on captions can be powerful enough for the scale of mPLUG-Owl (1000M+ text-image pairs). However, we believe our data is lightweight and effective, and our pipeline is more customizable.

On fragmented OCR results:

The reason why we tried to remove fragmented OCR results from prompting GPT-4 is to remove repetitive and meaningless questions generated by GPT-4. If there are only a few unrelated words presented to GPT-4, it is hard to generate meaningful question-answer pairs for the texts. Note that (1) our noisy instruction-following data based on raw OCR results still contain such fragmented OCR results. (2) Our experiments show that the learned text recognition capability transfers well to scenarios like ST-VQA and textVQA, where the texts in images are usually fragmented words.

Thank you for recognizing the practicality and effectiveness of our data collection and training methodologies, as well as the potential community benefits from the release of our data and model.

For methodology novelty, please take a look at the general responses.

Comparing to previous studies like PreSTU, Pix2Struct:

Our work focuses on improving the text recognition capability of the multimodal instruction-following model, which can be built on related prior work like PreSTU and Pix2Struct. Assuming we have a good frozen image encoder (which can be CLIP, Pix2Struct, or Pix2Struct), we study how to align its text recognition capability with large language models (pretraining stage) and how to maintain and acquire such capability during instruction-following (finetuning stage). In current Table 4 and Appendix E, we provide the results for using Pix2Struct to augment CLIP and demonstrate its improvement over CLIP $224^2$.

We assume the amount of data needed for feature alignment should be much less than that needed for feature learning. As our data and pipeline focus on feature alignment, we believe it can naturally benefit from any advanced image encoders like Pix2Struct and PreSTU.

GPT-4-based instruction-following evaluation

GPT-4’s responses are treated as oracles. We provide text-only GPT-4 with detailed descriptions of the image (human-written captions, OCR results) and collect feedback as oracles on related questions. To calculate the score, we provide text-only GPT-4 with the detailed description again, together with one question and two answers (one from text-only GPT-4, one from the model we want to test), and ask GPT-4 to give scores to the two answers (1 ~ 10). The final score is the ratio between the average score of the tested model and the average score of GPT-4. For example, “83.1” in Table 5 means its score is 83.1% of GPT-4’s score.

Thank you for recognizing our work's innovative aspects and potential impact! We address your comments below:

For comparison with baselines/methodology novelty/evaluation metric, please refer to the general responses.

Table of language model parameter size and training data size.

As all baseline models use ViT CLIP 224 as their visual encoder, we list the parameter size of their language model above, together with their training data size.

In our experiments, we find that the language model size has a trivial effect (see the table below) on the performance related to text-based VQA. We believe data efficiency is a more important factor to consider while comparing performance.

More comprehensive case study:

The case study is now conducted on 825 examples from OCR-VQA. We have updated Figure 6 in the draft. In this large-scale case study, we still observe a threshold for recognizable texts for both $224^2$-based and $336^2$-based LLaVAR as the accuracy sharply decreases when the height is smaller than 7 pixels.

On Figure 7:

We have provided an updated Figure 7 with detailed captions. A sketch of the implementation details is as follows: Given an image, it is simultaneously processed by visual encoders $V_1$ and $V_2$. $V_1$ features are transformed by transformation matrix $W$ and directly used as input embeddings to the language model. For $V_2$ features, they are transformed by transformation matrix $K$ and $V$ and used as keys and values to calculate the cross attention in every transformer layer (assume there are $N$ layers), which uses the transformed hidden states (through $Q$) from the self-attention module as queries.

The impact of OCR errors:

We take 1673 examples from OCR-VQA, which have ground truth answers with more than 10 characters, to study such OCR errors. We (i) define “correct” as the ground truth answers that are exactly in the predictions, and (ii) define “partially correct” as there exists a substring in the prediction that has high enough similarity with the ground truth but not the same. Specifically, we look at all substrings with the same length of the ground truth in the prediction to calculate ANLS (Average Normalized Levenshtein Similarity) and regard the prediction as “partially correct” if the highest ANLS is greater or equal to 0.5 but smaller than 1.

We find that a considerable amount of predictions can be considered partially correct, which indicates the actual performance of tested models is better than the reported accuracy numbers. However, the percentage of partially correct predictions is highly correlated with the percentage of correct predictions. Therefore, we believe that the current metrics can effectively compare the performance of different models.

Clarification on the term “temperature”:

The temperature used to sample examples from language models.