MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding

The paper lacks sufficient novelty, as there are already earlier works with similar approaches, such as MMLongBench-Doc. In the experimental section, the authors could conduct additional experiments for deeper analysis, such as incorporating human evaluation results and providing specific error case analysis.

1. Lack of discussion with existing document understanding benchmarks: The motivation of this paper is fine-grained visual understanding, with a specific focus on document understanding. However, I think the existing document understanding benchmarks such as DocVQA, InfoVQA [1], DUDE, TextVQA [2], ChartQA, MMLongBench-Doc [3] have already provided the fine-grained QAs. The QAs in these benchmarks have already required the fine-grained visual understanding ability of a specific region of the whole image. Many existing LVLM models, e.g., QWen2-VL also report their results on these existing document understanding benchmarks. The authors miss the discussion/citation with [1,2,3] and do not discuss the previous document understanding benchmarks in the related work section. What's the difference between MMDocBench and these existing document understanding benchmarks? I suggest the authors provide a comparative analysis table highlighting the key differences between MMDocBench and these existing benchmarks. This could include aspects like task types, annotation methods, evaluation metrics, and any novel features of MMDocBench.

   • [1] InfographicVQA, WCAV 2022
   • [2] Towards VQA models that can read, CVPR 2019
   • [3] MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations, NeurIPS 2024

2. About the LVLM baselines: I think document understanding and OCR are hot topics in LVLM research. The related work discussed in this manuscript is mainly 2023, and many new LVLM models are not discussed. In addition, I think some of the baseline models are not chosen properly. Please refer to 2.1 - 2.3 below for details:

   • 2.1: The authors lack the discussion or benchmark results with some existing LVLMs that are specially designed for document/figure/chart/OCR understanding, such as:
     • [4] TableLlama: Towards Open Large Generalist Models for Tables, NAACL 2024
     • [5] Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models, ECCV 2024
   • 2.2: In the related work section, the authors mention the LVLMs such as UReader and LLVAR that are focused on visual document understanding. Why do not report their results in Table 2?
   • 2.3: The selected proprietary models such as GPT-4o and GPT-4v are homogeneous, and the authors may consider other proprietary models such as Gemini-1.5 and Claude-3.5.

3. The IoU score results: According to Table 2 and lines 464-472, the authors report notably low IoU scores for all models, with GPT-4 achieving a mere 2.44% and the best-performing model reaching only 11.44%. This finding is perplexing, given the strong performance of these models on other metrics. Specifically, GPT-4, despite its low IoU score, achieves the highest EM score (66.40%) and the second-highest F1 score (68.87%). This discrepancy suggests that the IoU metric may not be the most suitable measure for evaluating region prediction in this context. I think It is unlikely that a model can consistently produce accurate answers without correctly identifying relevant regions.

   • Another possibility is that the author measured the IoU metric incorrectly. For example, the normalization of coordinates to the range [0, 1000] (lines 314-315) raises questions about the input provided to the LVLMs. But how did the models account for this coordinate normalization during region prediction? The varying image processing strategies employed by different LVLMs, such as the use of different tile sizes and resizing techniques, can impact the accuracy of coordinate transformation and region predictions. For example, 1) GPT-4o has two different image-processing strategies with different numbers of 512px tiles. 2) Some LVLM instruction following datasets (e.g., LLAVA-665k) use a different coordinate normalization method to the range [0, 1).
   • Additionally, Some LVLMs design special tokens (e.g., <|box_start|> and <|box_end|> for QWen2-VL) for referring grounding, do the authors consider that different LVLMs require different visual grounding prompt templates?
   • The authors claim that Since some LVLMs cannot follow the instructions well during the test, especially for the region predictions, we revise the instructions based on the setting of these LVLMs to improve their performance (Line 402), But the detailed prompts are missing and the implementation details are not clear. Do the authors think this step is fair for comparison?
   • These implementation details (input prompts, image processing strategies, special tokens, instruction following ability, and other facts that may impact the IoU score), while potentially significant, are not thoroughly discussed in the paper. The authors just claim that All LVLMs face significant challenges in predicting supporting regions. I think this claim is insufficient and lacks evidence.
   • I suggest the authors provide the following explanation:
     • Consider analyzing the relationship between IoU scores and answer accuracy (EM/F1-Score) to better understand this discrepancy.
     • Evaluate whether the IoU metric is the most appropriate measure for this task, or if alternative metrics might better capture region prediction performance. Does it is necessary to require the LVLMs to predict the box coordinates with the answers at the same time?
     • How the IoU metric was calculated and applied across different models.
     • Provide more implementation details, such as 1) how the coordinate normalization was communicated to the models during inference. 2) the potential impact of different image processing strategies on region prediction accuracy. 3) The details of the instruction following ability with different LVLM models and other variables that influence the results.

4. The box prediction results are not easier for readers to assess: The authors present the bounding box prediction results in Appendix A.4. It is hard for the reviewers to be aware of whether the box coordinates are correct or not. Can the authors provide visual examples with bounding boxes plotted on the images for a subset of representative cases? This would make it easier for readers to assess the accuracy of the predicted regions.

The differences between some document images and natural scene images require further analysis, as mentioned in the introduction. What does ‘’fine-grained‘’ mean? If it only refers to the answer in a certain part of the image, it seems to be no different from the existing evaluation benchmark (Chart QA, DocVQA, etc).

1. There are no experiments on the instruction sensitivity, while small modifications may cause large result differences.
2. The EM metric may not work in some scenarios since LVLMs's responses are diverse.
3. The analysis of experimental results should include the model architectures, training data, and model sizes to do a more in-depth analysis. How these factors affect the results.