SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding

W1: The authors should clarify whether the LoCAL-Bench dataset is necessary to demonstrate the effectiveness of the proposed method.

A: As described in our global response Q1, the primary motivation behind LoCAL-Bench is to demonstrate the multimodal understanding capabilities of models. The dataset includes questions designed to require the integration of both images and surrounding text for accurate answers, emphasizing multimodal reasoning rather than document length. Examples of such questions are provided in Appendix E.

W2: The generalization ability of the method remains untested, and lack of discussion on its performance on downstream tasks compared to other works.

A: We have cited TextMonkey in both the introduction and related work sections to provide context and acknowledge its contributions.

Our primary contribution is not to surpass large proprietary models like Claude-3 Opus or Gemini-1.5-Pro in overall capabilities, such as generalization ability, as smaller models naturally face capacity limitations compared to these larger models. Instead, our framework focuses on leveraging the base LMM’s existing generalization ability to improve long-document QA performance. By integrating a retrieval mechanism, our approach reduces distracting context in long inputs, allowing the base LMM to perform better on long-document QA compared to processing all pages directly.

Additionally, our QA module can be fine-tuned on specific QA datasets for enhanced task-specific performance or left unchanged to preserve the base LMM’s generalization ability. For instance, TextMonkey could also serve as the base LMM within our framework. The resulting LoCAL-TextMonkey model would enable scalable, efficient long-document QA while maintaining the original model’s generalization ability for single-page QA tasks.

W3: The method’s performance may decline when relevant evidence pages exceed the five-page limit, potentially affecting its general performance.

A: Thank you for highlighting this potential limitation of our framework. Our method is specifically designed to address non-summary QA tasks, where questions typically rely on localized information from a small number of pages. The choice to use the top 5 pages reflects a practical balance between 1. hardware constraints and 2. the requirements of widely used benchmarks:

1. As shown in Table 4, the memory cost of directly processing long documents remains a significant challenge for current LMM architectures, and our framework provides a practical and scalable solution to effectively handle long-document understanding within these limitations.

2. In the MMLongBench-Doc dataset, only 3.41% of questions require more than five evidence pages, and in the SlideVQA dataset, all questions require fewer than two evidence pages, suggesting that our framework is well-aligned with the requirements of these benchmarks and many real-world tasks.

W: The paper lacks a direct comparison between the LoCAL framework and Retrieval-Augmented Generation (RAG) methods using document parsers.

A: Thank you for pointing this out. The document parser (Adobe Extraction API, cited in section 4) extracts text from PDF files while ignoring images, making it unsuitable for scanned document image datasets and limiting its capability to support multimodal understanding. Additionally, the parser incurs a computational overhead of approximately 10 seconds per call, further reducing its practicality. As noted in our global response to Q1, we provided experimental results on MMLongBench-Doc and LoCAL-Bench, where text-only LLMs, such as GPT-4 and Phi-3 (the language model used in Phi-3-V and InternVL2), were used to answer questions based on parser-extracted text from retrieved pages. These results showed lower performance compared to corresponding LMM methods that directly process pages as scanned images. We will include these results and provide further discussion in the revised manuscript.

Q: How to evaluate the quality of the LoCAL-bench dataset

A: As noted in the global response Q2, LoCAL-Bench is a small dataset of 226 QA pairs, carefully reviewed by human annotators to ensure accuracy and consistency. GPT-4o was used primarily to filter the initial large collection of documents, selecting content suitable for creating questions that require both image and surrounding text to answer. We have included two examples in Appendix E to illustrate the dataset’s characteristics. The small size of the dataset allows for thorough human inspection, ensuring its quality is comparable to human-annotated benchmarks.

W1: This article needs to be compared with more methods, such as bge-large, NV-Embed-v2, SigLIP, ColPali.

A: Thank you for highlighting the missing methods in our evaluation. We have added the results for these methods in Table 1. Notably, ColPali is a specific instance of our framework. In Table 1, Col-Paligemma uses the same structure of ColPali and we used ColPali 1.0 checkpoint in this experiment. Despite these additions, our method consistently outperforms all others, further showcasing the advantage of our approach.

W2: Two LoRA with one model, actually there are still two models, not a real unified model.

A: Thank you for your valuable feedback. We acknowledge your point that using two LoRA adapters with a shared base model does not constitute a truly unified model.

Our primary objective is to optimize GPU memory usage by sharing the base model between two LoRA adapters, rather than merging tasks into a single parameter set. Since retrieval and QA are distinct steps in our pipeline, using two specialized adapters provides a straightforward and efficient solution, aligning with established practices in foundation models, such as the Apple Intelligence Foundation Language Models [1], where adapters are used for various tasks. While a unified model trained with both contrastive and next-word prediction losses could eliminate the need for one set of LoRA adapters, it would introduce additional training complexities and potentially compromise performance. Furthermore, retrieval and QA still need to be performed as separate steps to avoid the infeasible memory cost in processing more pages, as reported in Table 4.

[1]. Gunter, Tom, et al. "Apple intelligence foundation language models." arXiv preprint arXiv:2407.21075 (2024).

W: The paper could strengthen its contribution by clearly distinguishing how LoCAL improves upon existing methods like ColBERT and common LoRA-based adaptations in LMM retrieval and long-document understanding.

A: Thank you for your valuable feedback. We are grateful for the opportunity to clarify our contributions. Our approach provides an effective solution for long-document QA by unifying retrieval and QA tasks within a single framework, achieved by customizing a Large Multimodal Model (LMM) with LoRA adapters. This design eliminates the reliance on external retrieval systems, allowing the LMM to function as both the retrieval and QA module. By focusing on the selection of relevant pages rather than processing all pages simultaneously, our method offers a scalable and efficient solution for multi-page document understanding, aligning closely with real-world use cases. In comparison to ColBERT, which is limited to text-only inputs, our approach natively processes multimodal inputs. This avoids the inefficiencies associated with lossy and time-consuming OCR processes, resulting in superior retrieval performance, as detailed in Appendix F.

Q1: Has the addition of DocMatix-IR and PFLDocVQA to the original ColPali training data improved the retrieval module’s performance on Table 1 benchmarks and the ViDoRe benchmark?

A: We trained two versions of our retrieval module with the PaliGemma backbone: one using only ColPali training data and another using combined data from additional sources (DocMatix-IR and PFLDocVQA). Using GPT4o with a tailored prompt, we filtered out duplicate images to minimize potential overfitting and excluded unsuitable questions, such as:

1. Broad questions: Requiring summarization beyond a single page.
2. Non-specific questions: Not tied to image content (e.g., “What is the page number?”).
3. Cross-page reasoning questions: Requiring information from multiple pages.

The model trained with additional data achieved a ~1% improvement in top-1 accuracy on MMLongBench-Doc and a ~1.5% improvement in NDCG@5 on the ViDoRe benchmark, while maintaining comparable performance on other datasets. However, we were unable to reproduce the performance of ColPali 1.0 using released source code and training data on the ViDoRe leaderboard, possibly due to hyperparameter differences. Thus, we use the ColPali 1.0 checkpoint in our experiments as a stronger baseline.

Q2: How does the performance compare between BGE-M3 and SBERT?

A: Thank you for your question. We have reported the results for BGE-M3 in the global response Q3. Despite these additions, our method consistently outperforms all others, highlighting its strength. We have updated Table 1 of our manuscript to include these new results.

Q3: Add more baselines for the retrieval task.

A: We have added additional baselines for the retrieval task to provide a more comprehensive comparison. Furthermore, we have updated the CLIP results using the largest checkpoint and improved the results for our Col-Phi-3-V model with a more optimized checkpoint. Our method continues to achieve the best performance, further highlighting its advantages and robustness. These updates have been incorporated into the Table 1 of the revised manuscript.

Q1: Is LoCAL Bench necessary to demonstrate the effectiveness of the proposed method.

A: Thanks for the comments, which help us to realize the confusion in Section 4 regarding QA pair selection in LoCAL-Bench. We have included the updated description into the new version.

LoCAL-bench shows the necessity of using large multimodal models for document understanding over text-based models that rely on document parsers to extract text. Specifically, questions within the LoCAL-bench are filtered to exclude questions answerable with textual information only. Hence, all questions require both figures and their surrounding texts from the document to answer. Compared with LoCAL-Bench, most existing benchmark questions can be answered using extracted text.

We compare the QA performance of our method with text-only baselines that utilize the document parser on LoCAL-Bench and MMLongBench-Doc. We use GPT based evaluation as introduced in Section 5.2 and Appendix G. Our results show that multimodal models consistently outperform text-only baselines, with the gap being more pronounced on LoCAL-Bench, highlighting the dependency on both image and text. Using the retrieval module improved GPT-4o’s performance with image evidence. However, in the text-only setting, retrieval did not enhance GPT-4o’s performance, likely due to insufficient information in the evidence pages, where additional context could be beneficial.

Text-only QA:

Multi-Modal QA:

Q2: Quality and Ethics Concerns of the LoCAL Bench dataset

A: We thank the reviewers for highlighting this concern and would like to emphasize that creating synthetic datasets by crawling data from the web and leveraging models like GPT to generate datasets is a common practice in NLP research. LoCAL-Bench, derived from the web, has been curated and reduced to just 226 unique documents (as noted in section 4 data statistics). All QA pairs have been validated by human reviewers to ensure the exclusion of harmful contents and personal identifiable information. Additionally, we confirm that the licenses and usage terms of each document explicitly permit use for research purposes.

To address ethical and legal concerns, the benchmark does not distribute the actual documents but instead provides links to their original sources, thereby avoiding the replication of real files while preserving dataset integrity. Furthermore, the experimental results are presented only as aggregate statistics, ensuring no potential information leakage. This approach guarantees reproducibility while strictly adhering to ethical and legal standards.

We appreciate the ethics review committee’s feedback. In response, we have added an Ethics Statement at the end of the main text, incorporating the discussion from our global response to Q2 in the rebuttal. We acknowledge and accept full responsibility for any legal implications related to our collected benchmark. Additionally, we have integrated the necessary clarifications from the rebuttal into the camera-ready version to ensure transparency and alignment with ethical guidelines. Furthermore, we have renamed our proposed method and benchmark dataset to better reflect their functionality and use case.