Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding

We are grateful for the reviewer's insightful recognition of our extensive ablation studies and the novel relative text-position prediction task, which validate our approach's robustness. Below are our responses to the main concerns.

[W1] Robustness of the ablation studies

As 35.17% represents the performance of the moderate BLIP2-OPT backbone and 72.7% represents the performance of the larger mPLUG-Owl backbone on DocVQA, they are not directly comparable. To provide the proper baseline, we brought the numbers for the mPLUG-Owl baseline in Table D (row 5) from the ablation study of UReader paper [14]. To further address the concern about the robustness of our ablation studies, we have conducted additional experiments on the mPLUG-Owl-based model. Due to the limited time constraints in the rebuttal period, we focused on components that might be considered most marginal: the reconstruction process (row 2 vs row 4, row 6 vs row 8) and Relative Text-Position Prediction (RTPP) (row 3 vs row 4, row 7 vs row 8). Note that we compare RPT to RFT for RTPP experiments. As shown in Table D, the reconstruction loss and RPTT consistently improve performance on most tasks with both the moderate and large backbone models, BLIP2-OPT and mPLUG-Owl, respectively. We believe these consistent gains demonstrate the effectiveness of our components. We will provide the full ablation studies for the large model in the revised version.

Table D: Results of ablation studies on the reconstruction loss and RTPP. Note that we compare RPT to RFT for RTPP experiments. The last row for each backbone model represents our final model.

[W2] Analysis on using more scales

We appreciate the reviewer's suggestion to explore alternative configurations of our multi-scale feature pyramid hierarchy. Our method is designed to be flexible and can be extended to use an arbitrary number of scales. For example, a 3-scale configuration with feature hierarchy can be implemented as follows: combining features from scales 3 and 2, then merging the results with features from scale 1. Due to significant time constraints during the rebuttal period, it is challenging to conduct extensive experiments on these alternatives. We are currently running an experiment with the model that integrates 3 scales. We present the validation performance of each task after 2 epochs in Table E.

Table E. Validation performance with varying scales after 2 epochs.

When integrating the 3rd scale in addition to the 1st and 2nd scale, the number of visual inputs increases about $(1+g+4g+16g)/(1+g+4g) ≈ 4.2$ times, while the throughput of our model only decreases by a factor of 2. Note that $g$ represents the number of grids in line 124. The validation results demonstrate that incorporating the 3rd scale still helps the document understanding but the improvement gain is diminished. This is expected because, as more scales are added, the incremental benefit decreases since the model has already captured most of the relevant information from the initial scales. We expect this experiment to be finished within the discussion period and will post the performance on the test set.

[Q1] Clarification on Table 3

Yes. The results labeled as "Ours" in Table 3 indeed represent our final model, which includes the Relative Text-Position Prediction Task.

Dear Reviewer Pfd9

Because the end of discussion period is approaching, we kindly ask you whether our response is helpful to clarify you or not. Also, if you have any question or additional comments, please do not hesitate to contact us. We thank you for your time and efforts to review our paper.

Best wishes,

Authors.

Thanks for your timely response. As we stated in our rebuttal to Reviewer CDZQ, our main contribution lies not only in using multi-scale features, but also in how we aggregate them without significantly increasing computational costs. This is an important issue for MLLMs. While multi-scale features clearly benefit MLLMs, it is not computationally feasible without an appropriate strategy. By addressing this challenge, our approach enables broader applicability of MLLMs across different hardware environments.

Table F. Test scores of the BLIP-2-based model with varying scales.

The experiments with the 3-scale variant of our model have been completed and the test scores for the different number of scales across each benchmark are presented in Table F. We used the BLIP-2-based version of our model and all models are trained using RTPP for the text reading task for this experiment. The results show that incorporating an additional 3rd scale continues to help with document understanding, but the improvement gain becomes less significant. This is understandable, as the model has already captured most of the key information from the initial two scales, so the additional benefit is reduced. Note that training the model with all three scales without our hierarchical feature aggregation module is not feasible on A100 GPUs due to memory constraints. We will revise our manuscript to include this analysis.

We truly thank the reviewer for recognizing our key innovations in multi-scale feature fusion, novel instruction tuning, and comprehensive experimental results across different models. Below are our responses to the main concerns.

[W1] Comparison to the recent works

Table B: Comparison of different configurations over TextMonkey [R4] and mPLUG-Owl 1.5 [R5].

Initially, we compare the configurations with our framework and two other approaches, TextMonkey and mPLUG-Owl 1.5, represented in Table B. We would like to highlight two key differences between both models and ours. Both models use more advanced Multimodal Large Language Model (MLLM) backbones for initialization - Qwen-VL and mPLUG-Owl2. Our model, in contrast, is based on the mPLUG-Owl. Additionally, both models benefit from substantially larger training datasets and more trainable parameters, which directly impacts their performance.

Despite these advantages, according to the reported results, our method surpasses TextMonkey. Although the mPLUG-DocOwl 1.5 model performs slightly better than ours, this is possible given its use of a more advanced backbone and larger dataset. For a fair comparison, the best option would be to augment both backbones with our framework and train the models with the same data, which is not feasible within a limited rebuttal period. However, we expect improvements in our results by incorporating our framework into the new backbones with more data, since the multi-scale aggregation module facilitates efficient capturing of different levels of detail.

[W2] Effect of the reconstruction loss and RTPP on mPLUG-Owl

Table C: Results of ablation studies on the reconstruction loss and RTPP. Note that we compare RPT to RFT for RTPP experiments. The last rows for each backbone model represent our final model.

In response to concerns about the robustness of reconstruction loss and Relative Text-Position Prediction (RTPP) on the advanced backbone model, we conducted additional ablation studies on the mPLUG-Owl-based model, which are presented in Table C. Note that we compare RPT to RFT for RTPP experiments. The results demonstrate that both reconstruction loss and RTPP consistently improve performance on most tasks with both the moderate and large backbone models, BLIP2-OPT and mPLUG-Owl, respectively. We believe these consistent gains demonstrate the effectiveness of our components.

[Q1] Clarification on the receptive field

In line 131, the description “effectively enlarging the receptive field size” should have been removed. We are sorry for the confusion. To clarify, augmenting each grid by a 2 × 2 scale does not enlarge the receptive field as originally stated. Instead, this operation provides a more detailed view of each grid area by effectively zooming in, allowing our model to capture finer-grained visual features and smaller text fonts within each cell. This increased resolution of information potentially leads to a better understanding of local details. We will revise this description to accurately reflect the purpose and effect of this operation.

[Q2] Clarification of Matrix multiplication

We simply flatten both matrices to the shape of $(H_i \times W_i \times Q) \times C$. We will revise the description to provide a clearer explanation.

[R4] Liu et al., TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document, arXiv 2024

[R5] Hu et al., mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding, arXiv 2024

[R6] Ye et al., mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration, CVPR 2024

We are deeply grateful for the reviewer's insightful summary of our work, highlighting the key contributions of our multi-scale feature extraction approach, new instruction tuning tasks, and comprehensive experimental results. Below are our responses to the main concerns.

[W1] Incremental novelty (multi-scale features)

The proposed approach is the first practical solution for integrating multi-scale features in high-resolution multi-modal large language models (MLLMs), which was deemed impractical due to resource constraints. Our framework preserves the benefits of multi-scale features without substantial increases in computational costs. Specifically, we developed a hierarchical visual feature aggregation module that substantially reduces computational costs using cross-attentive pooling and feature pyramid hierarchy, making multi-scale integration computationally feasible for MLLMs. Unlike prior works relying on single-scale inputs, our method allows efficient multi-scale processing in MLLMs, significantly improving performance.

[W2] Justification on comparisons

We appreciate the reviewer's comments but respectfully and partly disagree that our comparisons are lacking, due to the following reasons:

Our research focuses on OCR-free multi-task learning with MLLMs for recognizing document images (e.g., DocVQA) and understanding natural images (e.g., TextVQA), which is different from task-specific learning and OCR-based learning. Direct comparisons with task-specific models would not fully capture this key aspect of our work. For instance, ChartPaLI [R1], the state-of-the-art for ChartQA, uses instruction data specifically designed for chart data; it doesn't align with our general-purpose approach.

On the other hand, the top performers on the referenced leaderboards, such as those for DocVQA and InfographicVQA, utilize significantly larger models, advanced language models with 20-55B parameters such as InternLM2 [R2] and PaLI-X [R3]. These are not directly comparable to our LLaMA-based mPLUG-Owl model due to significant differences in scale and computational costs. Moreover, the size and origin of the massive training data used for these large language models are often not fully disclosed, making direct comparisons potentially unfair. Hence, we compared our model only with OCR-free methods based on comparable MLLMs in terms of model size and dataset scale.

We included Donut as a baseline, a well-established OCR-free approach, to demonstrate the effectiveness of our approach; Donut has an advantage in document-related tasks because it is pretrained with document-specific data, but our method is still competitive without such a specialized pretaining technique.

[R1] Carbune et al., Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs, arXiv 2024

[R2] Cai et al., InternLM2 Technical Report, arXiv 2024

[R3] Chen et al., PaLI-X: On Scaling up a Multilingual Vision and Language Model, arXiv 2023

Dear Reviewer CDZQ

Because the end of discussion period is approaching, we kindly ask you whether our response is helpful to clarify you or not. Also, if you have any question or additional comments, please do not hesitate to contact us. We thank you for your time and efforts to review our paper.

Best wishes,

Authors.

Thank you for your thoughtful feedback and for recognizing the contribution of our hierarchical feature aggregation module. We appreciate your suggestion regarding the comparison with state of the art (SoTA) methods.

Table G: Comparison with task-specific SoTA methods.

We list the performance of state-of-the-art models at the moment of submission in Table G. The SoTA models can be categorized into 3 categories.

Large Foundation Models: These models leverage extensive pretraining datasets and large model parameters. For DocVQA and InfographicsVQA, InternVL-1.5-Plus [R7] ranks highest on the leaderboard. This model has 26B parameters and is pretrained on 26 dataset corpora before being fine-tuned on an additional 49 corpora. However, the exact amount of training data is not specified in their technical report. Additionally, QwenVL-Max is currently the SoTA for DocVQA, though technical details for this model are unavailable. For TextVQA and TextCaps, variants of PaLI [R8, R9] hold the top rank. PaLI, which is trained on the 10B-WebLI dataset, is a significantly stronger foundation model than our backbone, mPLUG-Owl, which only uses a 1B dataset for pretraining. While PaLI-3 [R9] has 5B parameters, it still benefits from the 10B-WebLI dataset for pretraining. Moreover, direct comparison becomes more challenging when considering the differences in the unimodal encoders of each model. InternVL-1.5-Plus consists of InternViT-6B and InternLM2-20B, while PaLI uses ViT-G/14 and UL-2 for vision and language processing, respectively. In contrast, our backbone, mPLUG-Owl, utilizes ViT-L/14 and LLaMA, which are considerably weaker than InternVL-1.5 and PaLI variants. Once again, we have compared our model only with OCR-free methods based on comparable MLLMs in terms of model size and dataset scale.

Methods Using OCR Engines: For VisualMRC, current OCR-free models lag behind LayoutT5 [R10], which utilizes external OCR engines. This is likely because VisualMRC data is text-rich and features long documents, indicating significant room for improvement. While LayoutT5 has a relatively small model size of 770M parameters compared to recent MLLMs, incorporating OCR engines in such tasks provides a substantial advantage, particularly in accurately processing and understanding extensive textual information.

Fine-Tuning Foundation Models with Document-Specific Data and Methods: Our work falls into this category. For DeepForm, KLC, WTQ, and TabFact, the current SoTA model is mPLUG-DocOwl 1.5 [R5]. As mentioned in our response to reviewer rxzX, mPLUG-DocOwl 1.5 is based on mPLUG-Owl 2, a direct extension of our backbone, mPLUG-Owl. While the models are similar in size, mPLUG-Owl 2 is known to perform significantly better than mPLUG-Owl. Additionally, mPLUG-DocOwl 1.5 is fine-tuned on 4M document datasets, which is several times more than our 650K dataset. It is also worth noting that mPLUG-DocOwl 1.5 was uploaded to arXiv in March and can be considered concurrent with our work. For ChartQA, where ChartPaLI is the SoTA [R1] as mentioned in the initial rebuttal, instruction data specifically designed for chart data is used, and the model is based on PaLI, pretrained with the 10B-WebLI dataset.

We hope that this response clarify you. If you have any question or additional comments, please do not hesitate to contact us.

[R1] Carbune et al., Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs, arXiv 2024

[R5] Hu et al., mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding, arXiv 2024

[R7] Chen et al., How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites, arXiv 2024

[R8] Wu et al., Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts, arXiv 2024

[R9] Chen et al., Pali-3 vision language models: Smaller, faster, stronger, arXiv 2023

[R10] Tanaka et al., VisualMRC: Machine reading comprehension on document images, AAAI 2021

We sincerely appreciate the reviewer's acknowledgment of our introduction of novel components, thorough ablation studies, and detailed model information. Below are our responses to the main concerns.

[W1] Effectiveness of reconstruction loss

Table A. Effectiveness of reconstruction loss on both models. The bold-faced numbers indicate the best performance in each column for each backbone model.

The reconstruction is performed only during training, so it incurs no additional cost for inference. Although the introduction of reconstruction loss increases training complexity, the extra cost is marginal, particularly compared to other components in our training strategies. More importantly, as shown in Table A from our ablation study, the reconstruction loss consistently improves performance on most tasks with both the moderate and large backbone models, BLIP2-OPT and mPLUG-Owl, respectively. We believe these consistent gains demonstrate the effectiveness of our reconstruction loss, especially given that the inference cost remains unchanged.

[W2 & Q1] Using Relative Text-Position Prediction (RTPP)

This comment including the associated question is a bit confusing partly due to the wrong terminology. It would be appreciated if you revise the comments for better answers. Doing the best under the current context, we presume that "Random Text Positional (RTP)" means Relative Text-Position Prediction (RTPP) in our approach. Using RTPP during inference is an interesting idea, but it may not be straightforward because it may require additional modules for generating proper instruction tasks or increase inference costs significantly; it can be a direction for the extension of our paper.

[W3] Lack of open source availability

We understand the concern about open-source availability. Due to the proprietary issue, we have to go through bureaucratic procedures to release the full source code. However, we will put our every effort into scientific transparency and reproducibility. We have already provided detailed implementation details in Section D in the Appendix. We will further provide extra details and ensure that our work is independently verified by the research community.

[Q2] More detail on Predicting Text Positional (PTP) task

The PTP task follows the same process as the Reading Partial Text (RTP) task for generating the position pair ($p_{start}$, $p_{end}$) and its corresponding text segment. The key difference lies in how to use this information:

1. RTP: The position pair is a part of the instruction, and the model predicts the text.
2. PTP: This is opposite to RTP: the text segment is given as the instruction, and the model predicts the position pair.

In both cases, the position is represented by ($p_{start}$, $p_{end}$), indicating the start and end positions of the text segment within the document. We hope this clarifies the encoding process. We are happy to provide more details or examples in the revised manuscript if needed.

[Q3] Table 3 Formatting Issue

We highlighted the best results among MLLM-based instruction tuning methods. We will clarify this in the revised version of our paper.

[Q4] Typos & Errors

Thank you for pointing out our mistakes and we will revise our manuscript for better clarity. FYI, RTPP stands for Relative Text-Position Prediction as specified in line 243, and consists of RPT (Reading Partial Text) and PTP (Predicting Text Position).

Dear Reviewer cWFY

Because the end of discussion period is approaching, we kindly ask you whether our response is helpful to clarify you or not. Also, if you have any question or additional comments, please do not hesitate to contact us. We thank you for your time and efforts to review our paper.

Best wishes,

Authors.

Comparison to recent SoTA methods

Reviewer R-CDZQ raised questions about our choice of baselines, particularly regarding the recent state-of-the-art (SoTA) methods from leaderboards for each specific task. For reviewer R-CDZQ, we have provided a detailed comparison and discussion of these SoTA methods, covering all relevant benchmarks. These SoTA methods typically consist of large foundation models or task-specific methods that may come with external OCR engines. However, it is important to clarify that our primary objective is not to develop a new foundation model from scratch, but rather to enhance existing MLLMs using document data. Additionally, our goal is to build a general document understanding model rather than focusing on specific tasks. Therefore, our comparisons have focused on OCR-free methods that utilize comparable MLLMs in terms of model size and dataset scale, or frameworks pretrained only with document images, such as Donut and Pix2Struct. Importantly, we demonstrate the effectiveness of our method by successfully enhancing two different foundation models (BLIP-2 and mPLUG-Owl), showcasing the versatility and broad applicability of our approach. We believe that direct comparisons with task-specific models and large foundation models would not fully capture the general-purpose nature of our approach.

Additionally, reviewer R-rxzX requested a comparison of our model with two recent concurrent works, TextMonkey and mPLUG-DocOwl 1.5, both of which were released on arXiv in March. We have included these comparisons in Table B, using statistics from their papers and official repositories. We adopt mPLUG-Owl as our backbone model whereas TextMonkey and mPLUG-DocOwl 1.5 utilize more advanced Multimodal Large Language Model (MLLM) backbones—Qwen-VL and mPLUG-Owl2, respectively—for initialization. Furthermore, both models benefit from significantly larger training datasets and more trainable parameters, which directly influence their performance. Despite these advantages, our method outperforms TextMonkey in the reported results. Although mPLUG-DocOwl 1.5 performs better than our model, this is likely due to its use of a more advanced backbone and a larger dataset.

We hope that our additional experiments and evaluations have addressed the concerns raised by reviewers and cleared some doubts regarding the proposed method. Thank you.