ADOPD: A Large-Scale Document Page Decomposition Dataset

We appreciate the reviewer for the positive feedback and valuable suggestions.

Q: Depth of Analysis

Thank you for your suggestion. We've addressed paper writing issues by thoroughly polishing our paper. This includes detailed descriptions of data collection, experimental results (models, data, parameters, and evaluation metrics), and a reorganized experiment section emphasizing key conclusions. We've also introduced additional experiments in the updated PDF.

Q: Comparative analysis

Thank you for your feedback. Following your suggestions, we conducted comparative testing on two document datasets: DocLayNet (Text Detection) and RVL-CDIP (Document Image Classification).

[1] DocLayNet

We investigated three settings: (a) ADoPD fine-tuning with zero-shot testing on DocLayNet. (b) ImageNet-based initialization followed by fine-tuning on DocLayNet. (c) ADoPD fine-tuned model initialization followed by fine-tuning on DocLayNet. The results are presented below (also updated in Table 4 (b)).

[2] RVL-CIDP

We compared CLIP vision backbones: (a) Laion pretrained backbone + RVL-CDIP fine-tuned classifier. (b) ADoPD +Doc2Seq pretrained CLIP backbone fine-tuned on RVL-CDIP (updated in Figure 6 (b) and Table 6).

Significant ADoPD-based improvements were observed across all types, which highlight ADoPD's advantages for finetuning or pretraining.

Q: Outlier Detection and Human-in-the-loop

Thank you for your suggestion. Below, we answer your question from two perspectives: outlier detection and human-in-the-loop:

[1] Outlier Detection

The document presents a challenge for feature extraction, combining both images and text. A potential drawback may be in feature extraction, particularly in the initial stages relying solely on CLIP trained on natural images. While introducing additional data enhances robustness, this remains an area for improvement. Additional outlier detection methods, such as textual features, will be explored in the Appendix.

[2] Human-in-the-loop

During the data selection, we found that sensitive images, such as those depicting violence, might present within the in-distribution. However, recognizing their undesirability, we deemed them as failure cases. Depending solely on outlier detection to identify such outliers might yield unpredictable results. Therefore, We opted for a human-in-the-loop approach for this process.

We engaged diverse annotators from various countries and genders to check selected data. Providing only basic keywords without explicit sensitive references, annotators were tasked with removing documents they deemed sensitive. Ultimately, we achieved a drop percentage of 1.77%, and manual checks confirmed the presence of sensitive information in these images.

We thank the reviewer for the positive feedback and suggestions.

Q: Cost of applying these different models

Good question! Our method involves multiple pretrained models. The costs (model and annotation) can be primarily categorized as follows:

[1] Data Filtering

Costs in this part include GPU and GPT-4 API calling expenses. In data selection, our challenge was identifying valid document images from the 175M Laion web image pool. We employed models like Document Finding Classifier, Watermark Detection, Language Detection, OCR tools, and CLIP zero-shot tagging. Except for CLIP, other models are lightweight. We utilize GPT-4 ($0.03/1k prompt tokens) for all data during collection and Doc2Seq & Doc2Tag, with costs limited to selected images.

[2] Data Annotation Cost

In our model-assisted data annotation approach, model inference and training occur for each batch of data sent to annotators, incurring relatively high GPU costs. However, this cost is still smaller than human labeling expenses. Additionally, periodic model refinement was observed to accelerate annotation efficiency and reduce costs. Refer to Fig. 12 (Appendix, Page 17) for the annotated cost, displaying a decrease with the increase in data.

Q: Unclear Section 4.1 and Reproduce

The section has been revised in the updated PDF. In addition, Sec. 3.3 provides detailed information on data collection and annotation processes, while Sec. 4.1 covers baseline model implementation, hyperparameters, and evaluation metrics. In Sec. 4.2, we analyze experimental results, including new experiments in Tables 4 & 6. We'll provide models and annotation guidelines for replication. The revised Appendix includes detailed data analyses and taxonomy analysis.

Q: With or Without Fine-tuning on ADoPD

Great question. in Table 4 (a), we compared models fine-tuned on EntitySeg and fine-tuned on ADoPD Doc2Mask. The zero-shot performance of EntitySeg-tuned models is lower than ADoPD-tuned ones, given EntitySeg lacks document data. Despite excelling in natural image segmentation according to Fig. 5 (a), EntitySeg-trained models are unfit for documents due to significant composition differences, where layout elements outweigh content significance.

Q: Other Document Datasets and Pre-training

In line with your suggestion, we conducted two experiments:

[1] DocLayNet

In Table 4, ADoPD-trained models achieve good recall on the DocLayNet test set, despite a low mAP. When comparing models fine-tuned on DocLayNet with pre-trained weights from ImageNet and ADoPD, the ADoPD-based models exhibit superior performance, emphasizing their generalization ability. Below, we present some results; for more details, please refer to our revised PDF.

[2] RVL-CDIP

We also conducted tests on RVL-CDIP, demonstrating a significant improvement when pre-training CLIP on ADoPD data and fine-tuning on RVL-CDIP, a dataset richer in data types than DocLayNet (refer to Table 6 and Fig. 6 (b)).

Q: Other issues (Caption Evaluation and Data)

We've enhanced caption evaluation in our revised paper. In Sec. 4.2, we used 5K test data to assess Doc2Seq's effectiveness. GPT-4 captions achieve commendable CIDEr scores, but a notable disparity persists compared to human annotations. Models fine-tuned on Doc2Seq match GPT-4 on B@n but exhibit significantly lower CIDEr scores. Additionally, in Fig. 6 (left), human-annotated captions are notably longer than machine-generated ones, influencing the evaluation results.

We release ADoPD to support foundational models in document understanding, aiming to inspire progress in document analysis.

We thank the reviewer for the encouraging words and helpful comments. We address concerns, questions, and comments below in detail.

Q: Mask Annotation Quality and Annotation Procedure

Great question. We've updated an explanation of our data annotation process in Sec. 3.3. Extracting box annotations from PDF is convenient. However, PDF extraction tools (e.g., PyMuPDF) may not fully parse structural information for certain PDFs, such as scanned documents in PDF format. ADoPD aims to avoid bias toward purely digital PDFs, as many documents are in image format.

For text detection, another reason we chose to have annotators label directly (instead of model-assisted) is that we observed that even when the model is trained with DocLayNet or DocBank, their prediction results introduce noisy errors and are not robust to real-world document images (especially visually-rich documents). This increases the cost, as annotators need to clean lots of redundant error bounding boxes. Therefore, we did not adopt DocBank's method of parsing digital PDFs but instead used human annotation. Meanwhile, we outline the cost per instance of our manual annotation. The cost of box annotation is low compared to mask annotation.

We can see that masks are the most time-consuming and costly part of the entire process. We present quantitative and visual results in Fig. 12, 13, 14 (Appendix, Page 17, 18, 19), demonstrating that as data annotation increases, the cost decreases when fine-tuning the mask prediction models for assistance. Additionally, we will release our data annotation guidelines.

Q: Captions and Quality

We address this question from two perspectives:

[1] Our caption is not just generated by BLIP-2

We apologize for any misunderstandings that may have arisen due to the writing issues in the previous article. We would like to highlight that our captions are not solely based on BLIP2. Instead, they result from Prompt-Guided Context-Aware Captioning (updated Sec. 3.2). In practice, our captions are rewritten by GPT-4 (Eq. 1&2) based on BLIP-2 captions, zero-shot labels (CLIP + Document taxonomy), OCR results, visual tags (using Recognize Anything Model), as well as rules. Additionally, in Sec. A.1.1 (Appendix, Page 14), we provide a detailed explanation of prompt design and conduct human evaluations to validate the effectiveness of our prompt designs.

[2] Prompt-Guided Context-Aware Captioning Benefits Vision-Language Modeling

We attempted to collect human captions but faced challenges in controlling their quality due to variations in annotators' understanding and summarization of documents stemming from diverse educational backgrounds. Hence, we only collected a human-annotated test set (5k) for evaluation, presented in Fig. 6 (left) and Table 5. We also find that GPT-4 rewritten captions can cover multimodal information of documents and summarize effectively with the combination of human-adjusted prompts and heuristic rules.

Q: Doc2Seq Data Collection and Splits

For Doc2seq data, we utilized prompt-guided context-aware captioning on all ADoPD images, resulting in 80k/20k/20k splits.

For Doc2seq model evaluation, we selectively annotated a subset of document images (5k) with human input. Table 5 employs human-written document captions for evaluation. Generally, BLIP/BLIP2 zero-shot captions exhibit lower quality compared to models refined with Doc2Seq data.

To further validate the effectiveness of ADoPD captions, we conducted the experiment in Table 6 (Please also refer to the response to Reviewer umJE regarding the validity of captions.). Specifically, we fine-tuned CLIP based on ADoPD captions, then fine-tuned only CLIP's vision backbone on RVL-CDIP. We observed a significant improvement in CLIP's performance through refinement with ADoPD on RVL-CDIP (see Fig. 6(b)). This indicates that rewriting document captions is better than using the noisy captions from Laion. Detailed analysis in Sec. 4.2 highlights the importance of caption rewriting for handling noisy data.

Q：Will the dataset be open-source?

Yes, the motivation behind creating ADoPD is to accelerate developments in document understanding. ADoPD represents a small step towards establishing a large-scale, high-quality document image dataset. There are still numerous directions to explore, including the analysis of multilingual documents and sensitive data. We are releasing ADoPD to support the development of future foundational models for document understanding.

We thank the reviewer for reviewing our paper and providing valuable suggestions.

Q: Background Setup and Technical Details

Thank you for pointing out the paper's writing issues. We've thoroughly polished it in the updated PDF, with key improvements highlighted below.

[1] Background Setup

Follow your suggestions, we added a document page decomposition task introduction at Sec. 1&3. To enhance clarity, Fig. 2 was moved to the Sec. 1 with refined words. In this adjustment, paragraph 4 aligns with Fig. 2 to facilitate better understanding.

[2] Technical Details

We have improved the writing of technical details.

• Sec. 3.2: We describe the models used in taxonomy discovery and improve the connection between Alg. 1 and the text by including Fig. 4 for better understanding. We add indices for each step corresponding to Fig. 4.
• Sec. 3.3: This new section now includes previously missing data collection details. We have also added detailed collection methods (e.g., prompts, costs, etc.) and data details in the Appendix.
• Sec. 4.1 & 4.2: We added a detailed description of the baseline models and evaluation. We supplemented Table 4 (a, b) and Table 6, along with Fig. 6, and included two additional analyses for generalization and captions.

Q: Contributions

The major contributions of this paper can be categorized into two main points: the collected dataset and the proposed methods for dataset collection and evaluation.

[1] Data Contributions

The main contribution is our extensive document dataset, detailed in Table 1. Notably, Doc2Mask and Doc2Box are human-annotated, while Doc2Seq and Doc2Tag involve pre-trained models (e.g., CLIP, GPT-4) with human assistance. Doc2Mask is the costliest part (as for costs, please refer to the annotation cost response to Reviewer aArm). Unlike natural images with fixed object sets, understanding and identifying areas to annotate in different document images requires prior comprehension.

We release the ADoPD to support foundational model development in document understanding, fostering advancements in document analysis.

[2] Method Contributions

Another contribution is our proposed data collection pipeline, detailed in Sec. 3.2 and Sec. 3.3. Data-driven document taxonomy discovery is a cornerstone, determining the diversity and distribution of our data. To the best of our knowledge, our paper is pioneering in leveraging large-scale pretrained models for taxonomy discovery and data filtering in the document domain.

Q: Model Evaluation and Data Analysis

Thank you for your suggestions. We answer this question from two aspects:

[1] Data Effectiveness

ADoPD stands out with fine-grained visual entity masks and text bounding boxes. Our experiments on data effectiveness focus on evaluating mainstream models, backbones (Table 2&3), and assessing the generalization ability of models trained on ADoPD.

In updated Sec. 4.2, we study the generalization ability with two new experiments: (a) Training models on ADoPD and testing on DocLayNet, results in Table 4 (b). (b) Testing ADoPD fine-tuned CLIP image encoder on RVL-CDIP, results in Fig. 6 (a) and Table 6. Below, we show the RVL-CDIP results, highlighting the benefits of our data.

From above results and Table 4,6, we can see that benefiting from the diverse and fine-grained dense annotations in ADoPD, the trained models exhibit good generalization across various document datasets (DocLayNet and RVL-CDIP).

[2] Data Quality

During data annotation, ensuring data diversity and quality is the original intention of ADoPD. For human-annotated data, we ensure that annotators and reviewers are separate to allow a fair confirmation of the quality and reliability of Doc2Mask and Doc2Box annotations.

Additionally, annotators from diverse backgrounds and countries were employed to assess the sensitivity of document images. With basic rules provided, they removed documents deemed sensitive. Data distribution details are in Fig. 9 (Appendix, Page 16).

We will provide detailed data quality statistical results (with human review) along with the annotation guidelines.