DocGenome: A Large Benchmark for Multi-Modal Language Models in Real-World Academic Document Understanding

W1. Concerns about type and difficulty of the questions generated by GPT-4V

Thank you for your observation. As detailed in the appendix, we initialized QA pair generation with GPT-4 using a diverse set of QA examples for few-shot prompting. Additionally, Moreover, our quality control team, comprising 20 individuals with PhD or Master’s degrees, not only corrects answers but also edits incorrect questions to ensure the quality and difficulty of the QA pairs.

W2. The evaluation metrics used for different tasks could be improved.

We also appreciate your suggestion, and we are indeed working on designing more reasonable evaluation metrics. We’ve provided the figure to include our ongoing research on a novel metric design for the Equation-to-LaTeX task, which evaluates both LaTeX text similarity and region matching between the predicted LaTeX-rendered equation and the original image.

Q1. More details about QA pairs generation and quality ensurance

In detail, the process is as follows:

1. GPT-4 was used to generate 7028 questions for 1757 paper samples.
2. Quality checkers first examine the questions, retaining or modifying them to obtain correct questions.
3. Each question was then allocated to two quality checkers for review and correction.
4. The checkers attempted to correct incorrect answers and assigned confidence scores.
5. Only QA pairs with the same answer and the highest confidence scores from both checkers were retained for the final dataset.

Finally, 2498 QA pairs were retained to form the QA test set, of which 1672 were modified by the quality checkers. The editing rate is 66.93%

We have already provided confidence score criteria in Appendix E.

Q2. General VLMs on the Equation-to-LaTeX and Table-to-LaTeX tasks We’ve included additional experiments with a general VLM on Eqaution-to-LaTeX and Table-to-LaTeX tasks. Taking Qwen2VL-7b as an example, its performance is not only inferior to the commercial closed-source model Mathpix but also falls short of our EqVLm and TableVLm trained on DocGenome.

• Eqaution-to-LaTeX:

• Table-to-LaTeX:

At last but not least, we would like to take this opportunity to thank the anonymous reviewer again for the insightful comments and valuable suggestions, which greatly helped to improve the technical quality and the presentation of this manuscript. We sincerely hope that our response has addressed your concerns.

W2. More insights/experiments on layout information and logical relationships

We deeply appreciate the reviewer’s thoughtful feedback and the time and effort dedicated to evaluating our paper. We propose the following insights and experiments:

• Better layout detection aids in document understanding: We have already included this part in Section 5.4 of our original paper. For questions related to figures or tables, we directly annotated the detection boxes for figures or tables on the document images to serve as the image input for the QA task. Taking internVL-1.5 as an example, when the images of the papers contain layout information (this information can be considered as a prompt) relevant to the questions, the performance of the QA task can be further enhanced.

• The relationships between document elements facilitate the expansion of multimodal data: Utilizing the relation information between entities, we can index associated modal information for specific tasks, enabling data re-annotation and expansion. For instance, in constructing table-related QA tasks, we can not only obtain images of tables but also index the text describing the tables in the document, thereby enriching multimodal information for re-annotation and task expansion. The relationships make our annotation information more flexible and actionable.

W3.  Concerns about variance and the structure of document due to LaTeX

We sincerely appreciate the reviewer’s comments and the valuable time spent reviewing our paper. As mentioned by the reviewer, in Table 6 (Line 458), Mathpix demonstrates slightly better performance on Out-Of-Distribution (OOD) data, compared to the model trained using our DocGenome, e.g., 0.4873 vs. 0.6627.

However, a direct comparison between our model and Mathpix may not be entirely fair, as Mathpix is a commercial software developed with an investment of hundreds of millions of dollars. In contrast, our DocGenome is an automated annotation pipeline that constructs datasets without incurring any cost. On the other hand, the diversity of our proposed DocGenome dataset in scientific literature is well ensured, thanks to the wide range of document layouts provided by the arXiv open-access community, which covers 153 academic disciplines. To validate this diversity, we conducted experiments from two key perspectives, as outlined below:

• Page-level diversity: Supplementary materials Figure A.1 demonstrate the diversity of the page-level data distribution.
• Layout diversity: Table 6 presents experiments conducted on a different domain, SciHub. The results demonstrate that models trained using the layout information from DocGenome outperform DocXChain and achieve significant improvements. This further highlights the diversity of the layout information in the DocGenome dataset.

Q1. More details in terms of the inter-annotator agreement or additional details.

In detail, the process is as follows:

1. GPT-4 was used to generate 7028 questions for 1757 paper samples.
2. Quality checkers first examine the questions, retaining or modifying them to obtain correct questions.
3. Each question was then allocated to two quality checkers for review and correction.
4. The checkers attempted to correct incorrect answers and assigned confidence scores.
5. Only QA pairs with the same answer and the highest confidence scores from both checkers were retained for the final dataset.

Finally, 2498 QA pairs were retained to form the QA test set, of which 1672 were modified by the quality checkers.

We have already provided confidence score criteria in Appendix E.

Thank you once again for your valuable feedback. We have revised our paper based on your suggestions, with all modifications highlighted in orange. Your input has been immensely helpful in enhancing the quality of our work.

We deeply appreciate the reviewer’s thoughtful feedback and the time and effort dedicated to evaluating our paper. We will address the reviewer’s concerns regarding the effectiveness of using LaTeX format in the following part.

LaTeX is the de facto standard for scientific writing, particularly in STEM fields, due to its ability to produce professional, high-quality documents. It is widely adopted by top journals and conferences, such as IEEE, ACM, and Springer. Note that many top journals and conferences require or recommend submitting manuscripts in LaTeX.

LaTex format has the following key advantages:

• Superior Mathematical Typesetting: LaTeX excels in handling complex equations, ensuring precision and aesthetic consistency.

• Extensibility: Thousands of packages (e.g., amsmath, graphicx, biblatex) allow customization for various academic needs.

• Graphics and Visualization: Tools like TikZ and PGFPlots enable seamless creation of high-quality, scalable graphics and plots. By leveraging the proposed DocGenome, we can extract the TiKZ code and the corresponding rendered image, as illustrated in this figure.

• Portability and Collaboration: As a plain-text system, LaTeX supports version control (e.g., Git) and works across platforms (Windows, macOS, Linux).

On the other hand, the arXiv community hosts papers under the CC license, and all papers are represented in LaTeX format. As a result, the structured scientific literature we obtain based on arXiv (using LaTeX code) also complies with the CC license, which helps to widely promote the dissemination and use of our open-source dataset (DocGenome). Moreover, by leveraging LaTeX code, we can automatically extract annotated structures from 600,000 scientific papers without incurring any human-annotation costs.

Overall, the collection is representative of scientific documents, as LaTeX is the preferred tool for academic writing in STEM fields due to its precision and professional formatting. While LaTeX does impose strict formatting rules, these are not constraints but rather mechanisms to ensure accuracy and consistency. For instance, the "??" marker for missing references serves as a clear indicator of errors, prompting authors to address them before finalizing the document. This feature actually enhances the quality of scientific writing by reducing the likelihood of overlooked mistakes. In contrast, other formatting software may not flag such issues, potentially leading to incomplete or inconsistent references. As a result, LaTeX’s strictness contributes to its reputation as a standard for rigorous scientific documentation.

At last but not least, we would like to take this opportunity to thank the anonymous reviewer again for the insightful comments and valuable suggestions, which greatly helped to improve the technical quality and the presentation of this manuscript. We have further supplemented some descriptive languages regarding LaTeX's representational ability in the main text. Hope our response has addressed your concerns.

It is appreciated that you could reconsider the representational ability and extensibility of our work, as well as our contributions to this community. Thank you.

Dear Reviewer TmKL,

Thank you for your continued engagement and for clarifying your concerns regarding our manuscript. We understand that your question centers on the representativeness of LaTeX-created articles across all scientific disciplines, rather than the importance of LaTeX itself.

We acknowledge that while arXiv covers a substantial number of fields, it does not encompass all areas of research, such as civil engineering, chemistry, and materials science. We appreciate your point that conferences and journals in these fields often offer both LaTeX and Word templates, which reflects the diversity of preferences within the scientific community.

To address your concern about representativeness, we would like to emphasize that our work primarily focuses on fields where LaTeX is predominantly used, as evidenced by its widespread adoption in journals and conferences within these domains. While our dataset may not cover every scientific discipline, it provides a comprehensive representation of those areas where LaTeX is the standard.

Regarding your question, "how representative LaTeX-created articles are," we have provided a specific implementation plan in the main text as follows:

• For disciplines not covered by the arXiv open-access community, such as civil engineering, chemistry, and materials science, we assume that we can collect their PDF data;

• We can leverage the models trained using our proposed DocGenome dataset to perform the inference processing on these out-of-distribution disciplines, such as civil engineering, chemistry, and materials science, thereby achieving generalization to these datasets;

• Actually, we have conducted additional experiments to evaluate the generalization ability of our approach to other disciplines. Specifically, we collected PDF data from the Scihub domain, which encompasses a broader range of scientific fields, including civil engineering, chemistry, and materials science not fully represented in arXiv. Using our DocGenome-trained models, we applied our scientific document parsing method to the Scihub domain. As shown in Table 6 of the main text, our method has demonstrated strong performance in scientific document parsing for Scihub domain, even when applied to diverse disciplines outside the core focus of arXiv. Notably, our approach outperforms existing layout detection tool, such as DocXChain, in terms of accuracy and robustness.

Table Caption: Results on Scihub domain, which shows the generalization ability of the proposed DocGenome on other disciplines outside the core focus of arXiv.

Finally, we would like to emphasize that we also recognize the value of diverse documentation tools and formats in scientific publishing. Our intention is not to diminish the importance of other formats but to highlight the strengths and capabilities of LaTeX in the contexts where it is most prevalent.

We have carefully revised our paper in response to your second-round questions and highlighted the changes in orange. Please refer to page 9 and Table 6. Looking forward to your response and further discussion.

Dear Reviewer TmKL,

Thank you for your further insightful questions. We appreciate the opportunity to clarify the details regarding the SciHub dataset and our comparison with DocXChain.

Q1. SciHub Dataset Details

We sampled the entire SciHub dataset and created a subset of 966 papers, covering disciplines such as medicine, chemistry, biology, and humanities as shown in the following table, in order to evaluate the generalization capability of our models across a wider spectrum of disciplines. While the exact distribution of fields within the SciHub dataset varies, it provides a diverse representation that complements the areas covered by arXiv. This figure visualizes some paper examples from our constructed SciHub dataset.

Regarding overlap with arXiv, only a very small portion of disciplines, such as computer science, intersect with arXiv's fields, and these account for less than 3% of the data. On the other hand, as you mentioned, our primary focus was on testing our approach in scientific fields where arXiv's coverage is highly limited, thereby highlighting the versatility and robustness of our document parsing method.

Table: The discipline distribution of the 966 papers from our constructed Scihub dataset

Q2. Comparison with DocXChain

DocXChain, a highly popular tool currently, is designed to handle a wide range of open-domain document parsing tasks. While it is a practical tool rather than a peer-reviewed research publication, our goal was to benchmark its performance against existing tools, demonstrating significant improvements in accuracy, particularly for papers outside the arXiv domain. This comparison is specifically designed to address the question of whether our approach can achieve a certain level of effectiveness, when applied to scientific documents from other disciplines. It is indeed evident from Table 6 of the main text, that our method can be applied to the Scihub domain.

We have supplemented the corresponding dataset information in the revised paper, and please refer to Table 7 in our main text. We hope this additional information addresses your questions and provides a clearer understanding of our work. We are grateful for your feedback, which has been invaluable in refining our manuscript.

Thank you once again for your thoughtful review and consideration.

 

Best regards,

Authors