M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework

1. How would direct text extraction and retrieval perform as an alternative to solve the problem? What are the performance on text-only questions? What are the performance on multimodal questions?

Thank you for raising this question. We assume that direct text extraction refers to using only the extracted text of each PDF document to generate the answer, without considering the visual contents of figures or tables. Regarding direct text extraction and retrieval, we separately consider text-only model inputs and text-only retrieval to avoid confounding analysis. Firstly, the results in Table 4, also attached below, show the effect of removing figure and table images from the model inputs, leaving only the extracted document text as input. Specifically, in this text-only setting, performance increases slightly for text-based questions, suggesting that for text-only questions, the visual content in the document may mislead the model in rare cases (lines 454-463). On the other hand, we found significantly lower scores in multimodal questions (figure-based and table-based), indicating that the direct text extraction alone cannot support multimodal questions well. Please let us know if you intended a different meaning of the question.

Regarding text-only retrieval, we compare the MRR score of different retrievers in Table 5 below. In this comparison, BM25 and BGE-M3 only consider document text for retrieval, while CLIP and ColPali can consider both text and images. We find that Colpali as a multimodal retriever significantly outperforms BGE-M3 as a text-only retriever, especially in figure-based and table-based questions. However, we note that text-only retrievers and multimodal retrievers tend to have different training data and model architectures specific to each setting. Thus, it may not be fair to make general conclusions regarding text-only compared to multimodal retrieval.

2. Retrieval tuning appears to yield only marginal performance gains. What specific challenges in retrieval are contributing to this limited improvement, and how do they align with the unique requirements of M-LongDoc?

Thank you for raising this question. To analyze the challenges in retrieval, we evaluated different retrieval settings in the table below. Based on the results, we believe the key challenge is the model’s ability to leverage relevant information from the retrieved pages (lines 359-362). To consider the ideal case of retrieval, we evaluate our finetuned model and ensure that the retrieved pages contain the gold evidence page (top-k=5 including gold page). In this case, we observe a slight improvement of +0.06 compared to standard retrieval with our finetuned model (top-k=5 retrieved pages). On the other hand, to consider the upper bound of model performance, we consider the oracle setting where the model is only provided the gold evidence page (no retrieval, gold page only). This results in a noticeable improvement of +0.20 compared to standard retrieval with our finetuned model.

Thus, while our tuning approach does show a significant improvement of +0.18 (4.6%) compared to the base model, there is also a significant gap to reach the upper bound performance with respect to retrieval. This suggests that distinguishing between relevant and irrelevant content in the retrieved pages in a key bottleneck. This challenge is core to the requirements of M-LongDoc, as our benchmark presents very long multimodal documents with hundreds of pages, which are not practical to fully process with large models. Thus, the ability to leverage retrieval and generate the correct answer based on the relevant pages is crucial to an effective document understanding system. We will include this analysis in the revised version, and aim to further study this challenge in future work.

1. All questions in the benchmark are synthetically generated by multimodal LLMs, which may limit the benchmark's reflection of real-world scenarios. Human annotations are not involved in the benchmark creation process.

While it is true that the questions in M-LongDoc were model-generated, we want to emphasize that all questions underwent rigorous human verification and quality control before inclusion in the final benchmark. Our team of expert annotators carefully reviewed each generated question to ensure relevance, appropriate content, and comprehensiveness. (lines 194-203). Furthermore, we believe that the generated questions are diverse and reflect real-world scenarios. Based on the example below from Figure 3, the question about oven vents is something a real person might ask when using or purchasing an appliance, related to everyday life and home cooking.

2. The dataset's scale is relatively modest (only 852 samples), potentially insufficient for capturing a wide range of perspectives and real-world scenarios.

While the M-LongDoc dataset contains 851 samples, which may seem modest, this size is comparable to other similar multimodal document understanding benchmarks, while requiring more in-depth answers. For example, MMLongBench-Doc contains 1051 samples. The controlled data size allows for careful human verification of each sample, ensuring high quality. Additionally, creating a larger dataset for long, complex documents with open-ended questions is extremely resource-intensive. We believe this approach balances scale and quality, providing sufficient samples to evaluate model capabilities while maintaining rigorous standards. The diverse document domains and question categories based on different multimodal elements provide a rich testbed, as shown in Table 1.

3. Evaluation relies on proprietary LLMs, introducing potential variability due to different checkpoints or versions.

While there may be some variability in different model versions, we have specified the fixed versions of gpt-4o-2024-05-13, claude-3-5-sonnet-20240620, and gemini-1.5-pro-002 in our paper for reproducibility (lines 407-420). The reason for selecting proprietary models is that they demonstrate leading performance (lines 241-243) compared to open-source models and are widely used in model-based evaluation, such as in “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment”.

4. Some related works are missing [1], the differences are not discussed. [1] DocBench: A Benchmark for Evaluating LLM-based Document Reading Systems

Thank for you mentioning this related work. While it is relevant, we also note that it was released on arXiv in July 2024, which is considered concurrent work according to ICLR guidelines. Regarding DocBench, it is similar to the existing MMLongBench-Doc as it also focuses on questions with short or extractive answers. In contrast, our benchmark mainly considers longer, open-ended answers which require more thorough understanding of the document. Furthermore, the authors of DocBench have found that multimodal models such as GPT-4o perform worse than text-only GPT-4, which indicates the benchmark may be less suitable for multimodal evaluation. On the other hand, our results from Table 4 below show that multimodal content is critical for our benchmark, as text-only inputs leads to significant performance degradation. We will include the discussion of this concurrent related work in the revised version.

We thank the reviewer for their positive review of our paper on M-LongDoc. Your feedback on the strengths of our work, particularly noting the usefulness of the new benchmark and training dataset and model for multimodal RAG-QA on long documents, is very encouraging.

We are pleased that you found the paper to be sound, well-presented, and a good contribution to the field. Your observation that the paper doesn't have any major weaknesses aligns with our efforts to create a valuable and robust contribution to multimodal document understanding. Your feedback reinforces our belief in the importance of developing resources and methods for handling long, complex multimodal documents. We see this work as a stepping stone for future research in this area, potentially inspiring new approaches to multimodal document understanding and retrieval-augmented generation.

1a. The scope of this paper limited the question only focusing on specific pages ignoring the more natural cases of answers distributed spanning pages.

Thank you for raising this point. We limited questions to single-page evidence for a more focused evaluation, requiring more comprehensive understanding of individual pages' contents before tackling multi-page reasoning. This approach enables the creation of a higher-quality dataset that is more straightforward to generate and evaluate at scale. While we also believe multi-page questions are an important future direction, the current setting remains challenging for open-source models and also reflects many real-world use cases. Overall, we believe the current setting addresses an important gap in multimodal long document understanding while laying the groundwork for more complex settings in future iterations.

1b. And the author mentioned the in-depth but there is no supporting analysis and results to show why the dataset shows more in-depth.

Based on the examples below which are included in our Figure 3, we show that questions in M-LongDoc are more complex than those from other benchmarks, as they requires a longer explanation or analysis rather than an extraction of a short text span or value (lines 125-130). Thus, we believe our benchmark is more in-depth as it requires more comprehensive and thorough understanding of each page in the document contents. This is further supported by the comparison on question lengths and answer lengths in the table below, showing that our questions and answers contain significantly more tokens on average. Thank you for raising this point, we will include this additional analysis in the revised version.

2. The dataset generation workflow uses off-the-shelf tools and models to extract the document structure which should be verified as the accumulated errors may occur when moving to the automatic QA generation stage.

While there may be some data extraction errors from off-the-shelf tools, we believe that our automated and human verification as discussed in Section 2.2 can address this concern. As each question addresses the specific text, table, or figure content in a document page, our verification also considers whether the specified data category is present in the extracted content, and whether the question can be answered based on the extracted content. Thus, each question is first automatically checked and then human verified to avoid errors from the data extraction tools. For reference, the full verification checklist is included in Appendix A.1 (lines 769 to 780).

3. More dataset analyses are expected including question length and focusing topics. Simple statistics can not show more insight of your datasets.

Based on the dataset, we find that the questions on average contain 31.6 tokens, which is significantly longer than other datasets, as shown in the table below. In addition to detailed statistics of the dataset, as shown in Figure 2, we illustrate the data topic distribution in Figure 1. The topics are obtained from the metadata of the documents, such as the paper category in the academic domain. As shown in Figure 1, the data domains span diverse topics such as machine learning, healthcare, and even laptop devices.

4. The proposed evaluation metrics may need more detailed analysis to show robustness. Current average weighting looks too simple ignoring the difference between specific models dealing with specific types of questions. Some penalty or reward terms may need to be considered.

While average weighting may look simple, we believe that this may be a strength as a simple metric is easier to understand, implement, and interpret. We observe that this is a general approach also adopted by previous works “Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models”. On the other hand, we did not propose specific penalty or reward terms as they may introduce unexpected biases based on different models. Additionally, we believe the current approach can robustly aggregate the judge scores, as we observed a high Pearson correlation of 88.9% between the scores from our evaluation framework and human scoring (lines 343 - 347).

5a. As the metrics are unexplored the results may not be comprehensive and reliable.

To assess the quality of our evaluation metric, we conducted manual human scoring based on the same evaluation guide, observing a high Pearson correlation of 88.9% between our evaluation framework scores and the human annotator (lines 343 - 347). Thus, we believe that the evaluation metric is shown to be reliable and with high agreement to human preferences. This is also supported by previous works in model-based evaluation, such as “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment”, which demonstrates that similar model-based metrics can achieve high agreement with human evaluators. We also feel that the evaluation metric is comprehensive, as the guide shown in Figure 5 and Appendix A.3 considers aspects such as accurate information, comprehensiveness, relevance, and coherence. Could the reviewer suggest any specific aspects for the evaluation to be more comprehensive?

5b. Lack of quantitative analysis show different domain, question types performance.

As shown in Table 2 and Table 3, we do include the evaluation results on separate domains (academic, product, finance) and question categories (text, figure, table). Based on the results, we provided analysis into the specific areas of improvement for multimodal models, such as the lower performance on table-based questions (lines 428 - 450). For example in Table 3, Qwen2-VL achieved a correctness of 4.08, 3.83, 3.62 on text-based, figure-based, and table-based questions respectively.

1. Does your dataset consider the spanning-page answer setting?

Currently we limit questions to single-page evidence to allow for a focused evaluation, establishing an important baseline before tackling multi-page reasoning. This approach enables the creation of a higher-quality dataset that is more straightforward to generate and evaluate at scale. While multi-page questions are an important future direction, the current setting remains challenging for open-source models and also reflects many real-world use cases. Overall, we believe the current setting addresses an existing gap in multimodal document understanding while laying the groundwork for more complex settings in future iterations.

2. Is there a dataset structure parsing quality checking procedure?

Yes, we leverage both automated and human verification to ensure that the multimodal content is correctly extracted and applicable to the question. For example, the verification includes whether the document content contains the specified category of text, figure, or table (194-203), and whether the question can be answered based on the extracted content. Thus if the content is not present due to document parsing failure or other issues, the sample would not be included in our dataset.

3. Is there any analysis or comparison before and after human checking automatically generated QA pairs?

As discussed in Section 2.2, we found that 80.1% of the generated questions satisfied the automated verification. Of these questions that passed automated verification, 80.9% also satisfied the human verification. Thus, we only retain 851 questions that satisfied both the automated and human verification (lines 203 - 205).

4. Why does the number of document pages per document look wired, especially for academic papers? It might be too long for an academic paper.

To collect multimodal documents in the academic domain, we sourced for thesis manuscripts in arXiv as they are longer in content. For the financial domain and product domain, we collect company annual reports and detailed product manuals respectively (lines 147-150).