LLaVA-Read: Enhancing Reading Ability of Multimodal Large Language Models

1. The innovativeness of the method is limited, as it is based on multiple visual encoder works, such as Mini-Gemini, mPLUG-DocOwl2, and others.
2. The method of integrating OCR is not very flexible, and the lack of OCR information can affect MLLM, for example, the accuracy and quantity of OCR.

It seems that the model's performance improvement largely stems from carefully designed pre-training tasks. How does the model's design regarding high and low resolutions aid in text understanding? What is the authors' perspective on this issue?

Some similar works, such as Ferret UI [a] and CogAgent [b], adopt similar ideas. Could the authors discuss the differences and advantages compared to them?

[a] You et al. https://arxiv.org/pdf/2404.05719 [b] Hong et al. https://arxiv.org/pdf/2312.08914

1. Explanation of Methodological Differences

• The paper could benefit from a more precise explanation of how the proposed methods differ from existing approaches. For instance, the Mixed Resolution Adaptor (MRA) appears similar to LLaVA-HR but has certain distinct elements. While previous methods fuse the final three layers, the proposed approach fuses the 1st, 4th, and 7th layers, with slight differences in notation. Additionally, it is still being determined whether the layout recovery process used in the OCR Tokenizer is identical to the existing method. Although sharing source code could clarify these issues, providing detailed explanations within the manuscript or appendix would help readers understand the unique contributions of this study and avoid potential novelty issues.

2. Benchmark and Performance Reproduction

Given that this study integrates multiple techniques from previous work, confirming their synergy through benchmarks is crucial. However, there are some concerns regarding performance evaluation:

• The main performance results in Table 1 differ from the values reported in prior studies, with some benchmarked models performing lower than expected. This discrepancy calls for a more transparent explanation of the reproduction process. For example, DocVQA scores for EAGLE (Shi et al., 2024) are reported as 86.6 (Llama3-8b) and 85.4 (Vicuna-13B), but in this study, the reproduced score is 68.5, indicating a significant gap. Similar trends are observed with Idefics-2 and others. Although exact reproduction is challenging due to differences in model architecture, pretraining and fine-tuning data, and training methods, a clear explanation is more significant since the study emphasizes the high performance of the proposed model.
• The table should clarify model names for the alternatives. To ensure fair comparisons, models with smaller parameters than the proposed approach's Vicuna-13B should be labeled accordingly (e.g., Idefics-2 at 8B and EAGLE at 8B and 13B).

3. Coverage of Related Research

The paper would benefit from including additional key studies in related work:

• The use of external OCR tools, as employed in CREAM (Kim et al., 2023). Multiple Vision Encoders have been used in models like the SPHINX series (Lin et al., 2023) and DeepSeek-VL (Lu et al., 2024).
• Essential works in text-rich document understanding, such as UDOP (Tang et al., 2023) and Idefics3 (Hugo et al., 2024), should also be referenced.
• Although recent papers like Llama-3.2 (META, 2024) and Qwen2-VL (Peng et al., 2024) may not have been available during submission, they have achieved notable performance metrics and could be referenced in future work.

Including these studies in the Related Work section would enrich the paper's context.

4. Ablation Study Clarifications

There are some areas where the ablation study could be improved:

• The apparent effect of layout recovery has yet to be fully demonstrated. A baseline comparison with methods like layoutLM or pix2struct would be beneficial. If the LATIN-Prompt (Wang et al., 2023a) method is used exactly as in prior work, a mention of its verified effectiveness in those studies will clarify this.
• Providing additional performance metrics for ablation would help, as the current Table 3(b) lacks "recognition score" and "total". Including these and more tasks from Table 1 beyond OCRBench would strengthen the ablation analysis.
• The ablation studies should ideally fix the proposed model's resolution. Since most ablation studies in Table 3 are conducted at 1024 resolution, this may be the default resolution. Also, confirming whether the LLaVA-Read results in Table 1 are all based on 1024 resolution would add clarity.
• Lastly, if the two vision encoders differ from those in LLaVA-HR, explaining the criteria used for selecting them would enhance the transparency of the model's design choices.