BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks

We sincerely thank the reviewer for their thorough and constructive feedback that will help strengthen our work. We appreciate the recognition of our key contributions in addressing licensing restrictions, ensuring data quality, providing comprehensive task coverage, and maintaining open-source commitment.

We appreciate the raised concerns and will address and clarify each of them as follows.

W1: (a, b): Human Verification of Data Creation in BigDocsBench

To address the concerns about verification methodology and evaluation criteria for BigDocs-Bench, we provide the following details:

A. Number of Human Verifiers and Their Qualifications

We engaged a team of 6 human verifiers, consisting of graduate-level researchers and professionals with expertise in multimodal datasets and document analysis. All human verifiers were trained on verification criteria mentioned below before beginning the review process. We have at least 2 verifiers on each data sample.

B. Verification Methodology

Automatic Filtering Tools: We implemented multiple automatic filtering tools as the first step in our quality control process. These tools were designed to detect issues such as bad annotations, NSFW content (using Llama-Guard-3 from https://huggingface.co/meta-llama/Llama-Guard-3-11B-Vision), and Personally Identifiable Information (PII). This automated process reduced the volume of problematic samples before any human verification was performed.

Human Verification for Test Split: To ensure a high-quality test split, we only do human verification focused on the test set as our automatic filtering is already quite robust. Annotators identified and removed problematic samples following the guideline, for example, unnecessary comments in LaTeX and HTML code (e.g., in Table2LaTeX and Screenshot2HTML tasks), which could cause rendering issues. Similarly, flowcharts with excessive isolated nodes in tasks like Image2Flow were flagged and removed. For Image2SVG and three GUI-related datasets, we spotted and removed those images with PII or NSFW content. All these typical errors were collected and documented during this process to inform our automatic filtering tools.

Refining Automatic Filtering Tools for Training Split: The typical errors identified during human verification of the test set were used to iteratively improve our automatic filtering tools. This enhancement enabled the tools to better detect and handle similar issues in the training split, which was impractical to verify comprehensively through human labor.

Sampling-Based Human Verification for Training Split: To ensure the training split met quality standards, we randomly sampled 100 samples from all tasks, proportional to the size of their respective training splits. These samples were verified by annotators based on the criteria mentioned below, and the overall pass rate was 99% (i.e. 99 good samples), reflecting a high level of quality for the training data after applying the refined filtering tools. Specifically, every task has a pass rate of 100% except GUI-VQA, which has a pass rate of 92% (i.e. 1 bad sample out of 13, which is not relevant to the task since the question is not quite related to the given image).

C. Verification Criteria For Verifiers

The verification process for BigDocs-Bench prioritized several critical aspects to ensure the dataset's quality, reliability, and suitability for real-world applications.

First, data integrity was a primary focus, ensuring accurate alignment between inputs and outputs. For example, in tasks like Image2SVG, visual elements in images were verified to match their corresponding SVG annotations, ensuring precise vector representation. Similarly, for Screenshot2HTML, rendered HTML outputs were checked against the screenshots to confirm structural and semantic fidelity.

Second, the relevance of synthetic samples was thoroughly assessed to confirm that they were realistic and reflective of the intended task objectives. For instance, in the Image2Flow task, flowchart samples were evaluated to ensure that the generated JSON or GraphViz code meaningfully represented the logical flow of the diagrams without excessive isolated nodes or disconnected components, which could compromise their utility. Finally, to ensure content safety, NSFW content and PII, such as explicit material, or personal data, were removed.

D. Inter-Rater Reliability

In the test split of our benchmark, we prioritized recall to ensure no potentially problematic samples were missed. For tasks like filtering NSFW content, detecting annotation errors, or identifying corrupted data, any sample flagged as problematic by a single verifier was removed. This conservative recall-focused approach maximized dataset quality and safety while addressing misalignment between raters.

We have added this procedure in Appendix A.13.

Dear Reviewer,

Thank you for dedicating your time and effort to reviewing our manuscript and for providing insightful suggestions. As the author-reviewer discussion phase nears its conclusion, we would like to ensure that our responses have adequately addressed your concerns. Please feel free to reach out if you have any additional questions or require further clarification.

Best regards

We appreciate the reviewer’s positive assessment of our dataset and benchmark contributions, the comprehensive experiments, and the inclusion of human evaluations. We are glad that the permissive licensing, breadth of experiments, and the overall utility of BigDocs-7.5M and BigDocs-Bench were seen as valuable contributions.

We respond below to address the concerns raised in the review.

W1: GPT-4o and Claude-3.5-sonnet underperform on BigDocs-Bench (Table 3)

We appreciate the reviewer’s observation about the variability in model performance across tasks and acknowledge that this highlighted an area where our original methodology could be refined.

Explanation of Initial Results

The performance discrepancies noted by the reviewer stem from differences in model training, instruction tuning datasets, and prompt templates. Each model requires specific prompt engineering to produce optimal outputs, as their responses vary widely in style and structure. For instance, some models adopt a conversational tone, while others are more structured, making uniform evaluation challenging.

In our original submission, we applied a single, consistent prompt across all closed models for fairness. However, we acknowledge that this approach limited the ability of certain models to fully showcase their strengths, particularly on complex tasks like Chart2MD and GUI2Sum. Parsing the outputs was also challenging due to the heterogeneity in formatting, and this likely impacted the results for some tasks.

Improvements Made

We conducted a thorough review and optimized prompts for each model to better align with their strengths and output styles. We found that some models needed prompts like "answer briefly" or "answer in one sentence or phrase" to avoid long, chatty outputs, especially for tasks like Chart2MD, which involves markdown generation. For tasks like Image2Flow, we found it crucial to use 1-shot prompting by providing the format of a training example. This helped models understand formats like JSON or Graphviz, which were specified in the prompt but not always generated correctly by most models.

We have re-computed all benchmark scores with the new prompting framework, and have updated the corresponding Table 3 in the paper. We provide here an image of the new table, with the edits highlighlted https://imgur.com/a/yKT7VAB.

We are pleased to report that this optimization has led to a significant performance boost. For Chart2MD, Claude3.5-Sonet saw an improvement of 54%, while GUI2Intent improved considerably, from 4.87% to 13.12% with Claude3.5-Sonet.

The updated results now show a more consistent and expected pattern, with state-of-the-art closed models outperforming open-source models by about 30%. BigDocs models continue to lead with a strong advantage of 40-50%, reaching 56.32% in the best case with Phi3.5-Vision.

We appreciate the reviewer’s feedback, which prompted us to revisit and improve our evaluation methodology. We have updated the paper's tables and discussion to reflect the revised benchmarks, ensuring a more accurate representation of model performance.

W2: Correlation analysis between BigDocs-Bench and other benchmarks

We appreciate the suggestion of a correlation analysis to highlight BigDocs-Bench's uniqueness in the VLM space and thank the reviewer for recommending benchmarks such as RULER [1] and "human-eval" (likely referring to HumanEval [2]). These datasets, however, might not be suitable for correlation analysis in our vision-language model (VLM) setup, mainly because they are text-only benchmarks. RULER [1] is an LLM benchmark designed to measure long-context performance, while HumanEval [2] is a benchmark to evaluate models on code generation tasks. Both are text-only benchmarks.

While the proposed benchmarks may not directly apply to our multimodal setting, we acknowledge the importance of such correlation analysis. We thus performed a study on the correlation of BigDocs-Bench and other popular general VLM benchmarks. We present the results in the form of a correlation matrix, a dendrogram plot, and a PCA analysis. We facilitate the figures here (https://imgur.com/a/SQHV3lm), and we also added them to the manuscript (Figures 16 and 17)

Please see our response to reviewer gUaq (W2), who also requested a correlation analysis focused on the document benchmarks presented in the original paper. This might be of interest to the reviewer for a more detailed view of document-specific benchmarks.

Analysis Setup. We selected all benchmarks listed in Tables 2 and 3, covering both General Document Benchmarks and our proposed BigDocs-Bench, and expanded the analysis to include 28 additional VLM benchmarks (see figure for the complete list). The matrix was constructed using all models presented in the paper, including instruction-tuned versions of DocOwl-1.5-8B, Qwen2-VL-2B, LLaVA-NeXT-7B, Phi3.5-v-4B, Qwen2-VL-72B, Llama-3.2-90B, GPT-4o, Claude 3.5 Sonet, Qwen2-VL-72B, and GeminiPro-1.5. Notably, we added Llama-3.2-90B to our baselines during the rebuttal period.

We evaluated as many models and benchmarks as resources allowed to construct a complete matrix of benchmarks and datasets. Missing scores were imputed using the mean to ensure a fully populated matrix for the analysis.

Methods.

1. Cosine similarity matrix and dendrograms: We normalize metric scores by centering (subtracting the mean) and scaling (dividing by the Euclidean norm). We then apply hierarchical clustering using cosine distance to group benchmarks and visualize their relationships. Cosine similarity matrices and dendrograms illustrate these clusters.
2. Principal Components Analysis (PCA): We represent each benchmark as a feature array composed of scores from all models and project these arrays onto the two principal dimensions.

Results

Figure 16 (https://imgur.com/a/SQHV3lm, top) presents the correlation matrix and dendrograms. Circles represent BigDocs-Bench tasks (our benchmarks), upward triangles denote other document benchmarks included in the paper, and downward triangles denote general VLM benchmarks. BigDocs-Bench tasks demonstrate distinctive evaluation characteristics compared to existing benchmarks in the VLM space. This is evidenced by the clear clustering pattern, where BigDocs-Bench tasks form their own correlated group while showing minimal correlation with other benchmark types. Results also reveal three main clusters of tasks: general vision-language tasks (like COCO_VAL and VCR), document understanding tasks, and GUI/table-related tasks (such as our tasks GUI2Sum and Table2Latex). This clustering pattern suggests that BigDocs-Bench is addressing unique aspects of document understanding that is not captured by existing benchmarks.

Figure 17 (bottom) illustrates the PCA results, revealing that most BigDocs-Bench tasks are clustered in the bottom-right corner, distinctly separated from other benchmarks. Notably, tasks within BigDocs-Bench form clear clusters based on their type, such as VQA, Flows, Charts, or GUIs. General VLM benchmarks create a dense cluster, while document benchmarks are more sparse.

Unique Aspects of BigDocs-Bench. Our correlation results show a clear separation between BigDocs-Bench and the larger VLM benchmark landscape. In summary, it introduces the following novel elements to the landscapes of VLM evaluation and document understanding:

1. Document and Chart Reasoning: Tasks focused on understanding, question answering, and summarization of charts, graphs, and workflows from images.

2. Web and GUI Understanding: Tasks targeting web and graphical user interface comprehension, a less explored area in current benchmarks.

3. Multimodal Code Generation: Tasks involving the conversion of images into code representations, such as inverse rendering and code generation (e.g., HTML, SVG, JSON).

4. Long-Context Structured Output Generation: Tasks requiring structured output over extended contexts, such as generating long JSON, SVG, HTML or Markdown from images

References:

[1] Hsieh, Cheng-Ping, et al. "RULER: What's the Real Context Size of Your Long-Context Language Models?." arXiv preprint arXiv:2404.06654 (2024).

[2] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

Q1: "The context length of the models (8192 tokens) might impact performance"

This is a valuable observation, particularly given that the Screenshot2HTML task, which averages around 32k tokens per example(as shown in Table 1), faces significant challenges under an 8k-token limit. This task ranks among those with the lowest scores (as seen in Table 3), contributing to a decrease in the overall aggregated average score. As such, we believe that *the Screenshot2SVG task is could be affected by saturation due to the limited context length. Therefore, we focus our answer on this task.

Despite this limitation, we observed that the generated HTML code remains compilable, and the resulting DOM can still be evaluated using the DOM Tree Edit Distance metric, even though the HTML output is truncated due to the context limit. However, the inability to handle complete HTML outputs underscores the need for models capable of processing extended context lengths.

To better understand this issue, we analyzed the model outputs (HTML code) in terms of a histogram of token lengths, comparing them with the ground truth lengths in the test set (both images are available here: https://imgur.com/a/9oLDAlA). It’s important to note that the test set is designed so that the images can be represented in fewer than 8192 tokens (top figure). This analysis confirms that the task is indeed suffering from token length constraints, as most samples reach the maximum token length of 6k (after subtracting image tokens), as seen in the bottom figure (https://imgur.com/a/9oLDAlA). Although the test set is intended to be generated within a reasonable token length, the model attempts to generate longer HTML outputs due to its training on larger contexts.

We are conducting an ablation experiment on the context length by fine-tuning the Phi-3.5-Vision model with an extended context length of 16k. This training is still ongoing, and we will share the results once available.

There are several key reasons for our decision to fix the context length at 8192 tokens:

1. Model Compatibility: Not all models evaluated in our benchmark natively support larger context lengths, and extending this limit could disadvantage some models.

2. Compute Efficiency: Increasing context lengths substantially boosts GPU memory usage, making both training and inference at scale more challenging.

3. Comparability: To ensure fair comparisons across all models, we standardized the context length at 8k tokens.

In conclusion, the reviewer’s observation effectively identified the issue with the token length constraint caused by the 8k token limit in the Screenshot2HTML task. Training a model with a larger context and re-evaluating may yield improvements for this task. Nevertheless, this highlights the importance of long-context tasks, such as HTML generation, as challenging benchmarks that drive innovation.

We appreciate the thoughtful review and the recognition of the contributions made by BigDocs-7.5M and BigDocs-Bench. We are pleased that the reviewer acknowledged BigDocs-7.5M's novelty, openness, and performance benefits, as well as the robustness of BigDocs-Bench for diverse multimodal evaluations.

We also appreciate the reviewers' concerns and are glad to address them.

W1: "More clarity on human evaluators"

To ensure fairness and minimize bias, we invited evaluators from diverse backgrounds, including PhD researchers, ML practitioners, multimodal AI experts, and individuals from both technical and non-technical domains. Key authors were excluded from participating to avoid conflicts of interest. For the evaluation, we used our own web interface, to make sure that all participants were anonymous and model outputs were randomized to prevent pattern recognition. We are open to releasing the detailed human evaluation results, as a supplementary document.

We sincerely thank the reviewer for their thoughtful comments and constructive feedback. We appreciate the recognition of the dataset’s license permissiveness, data curation, multimodal considerations, and data contamination analysis. These were deliberate design decisions to ensure that BigDocs-7.5M and BigDocs-Bench contribute meaningfully to the research community.

Below, we address each concern and clarify the aspects mentioned, incorporating additional analyses to the manuscript.

W1: The effect of OCR, VQA format, and text-image alignment is not demonstrated in the experiments.

We appreciate the reviewer’s observation regarding the need for clarity on the impact of various data curation processes. To address this, we conducted an ablation study to isolate the effects of each processing step.

Experiment Setup. Using the Phi3.5-Vision-Instruct model, we created three modified versions of BigDocs-7.5M: one without the VQA format, one reducing OCR tasks to 30%, and one reducing text-alignment tasks to 30%. Each dataset variant was used to train the model, followed by fine-tuning the model on downstream tasks. We evaluated these models on the benchmarks listed in Table 2 (General Document Benchmarks), with results shown below, including the differences from the original results in parentheses.

Main Findings

1. VQA formatting was critical, with its removal leading to a 9.19% average performance drop, especially on InfoVQA (-15.13), DeepForm (-30.33), WTQ (-14.28), and TabFact (-15.94). A common issue observed is the failure to follow the requested output format, as shown in Table 26 (see https://imgur.com/a/08H5tDH). We hypothesize that the VQA format used during continual pre-training (CPT) improves instruction tuning and supports multitask generalization.

2. Removing OCR tasks caused a 2.71% average performance drop, with significant declines on InfoVQA (-10.02), DeepForm (-16.78), and TabFact (-7.48), while DUDE, SlideVQA, and ChartQA remained largely unaffected. Reducing OCR data increased hallucination in tasks requiring precise information extraction. In DeepForm, for example, the true negative rate dropped from 62% to 0%, with models hallucinating unrelated values from other document sections. Examples of this behavior are shown in Tables 21 and 22 (https://imgur.com/a/bPomLGE). In some cases, such as TableVQA (+3.06) and TextVQA (+3.35), performance improved, likely due to the advantage of free-form text generation over exact OCR matches. (See example: https://imgur.com/a/jN8V84z)

3. Reducing text-image alignment tasks, especially captioning, caused a 3.43% performance drop, with significant declines on InfoVQA (-12.07), DeepForm (-17.61), and TabFact (-8.74). This drop was mainly due to errors in parsing values for arithmetic operations, especially when converting text or visual elements into percentages or performing calculations on extracted visual data. This is shown in Tables 23, 24, and 25 (see example here: https://imgur.com/a/6Oqy6RC). Tasks like WTQ and ChartQA, which involve simpler text-visual correspondences, do not require complex conversions and thus show no significant drops.

These results highlight the importance of VQA format, OCR, and text-alignment tasks, and suggest the potential for optimal data mixtures across tasks, datasets, and sample sizes. We are experimenting with different weight mixtures of BigDocs-15M and will share the results if they provide valuable insights. We appreciate the reviewer’s suggestion, which led to this analysis, now included in Table 7 (Appendix A.12) of the updated manuscript.

W2: “How does the aggregated score on BigDocs-Bench correlate with existing benchmarks?”

We thank the reviewer for raising this important question. To better contextualize BigDocs-Bench within the broader VLM and Document Understanding research landscape, we conducted a correlation analysis, comparing BigDocs-Bench scores with those of other document benchmarks presented in our paper. This evaluation includes both aggregated scores and task-level performance. We present the results in the form of a correlation matrix, a dendrogram plot, and a PCA analysis. We facilitate the figures here: https://imgur.com/a/jYR57Jz

We also recommend the reviewer refer to our response to Reviewer UxFn (Response W2), which addresses the same issue and provides additional insights on broader VLM benchmarks.

Analysis Setup. We selected all benchmarks listed in Tables 2 and 3, encompassing both General Document Benchmarks and our proposed BigDocs-Bench, respectively. We constructed a matrix encompassing all models presented in the paper, namely DocOwl-1.5-8B, Qwen2-VL-2B, LLaVA-NeXT-7B, Phi3.5-v-4B, both the off-the-shelf public versions and the ones trained on BigDocs. We also extended Table 2 with scores for GPT4o, Claude 3.5 Sonet, Qwen2-VL-72B, GeminiPro-1.5, and Llama-3.2-90B, to obtain a full matrix. Note that we have now extended our baselines with Llama-3.2-90B. Finally, we aggregated results for all metrics in the two groups of benchmarks. We have edited our manuscript to include Table 8, which is an extension of Table 2 with scores from all models.

Methods.

1. Cosine similarity matrix and dendrograms: We normalize metric scores by centering (subtracting the mean) and scaling (dividing by the Euclidean norm). We then apply hierarchical clustering using cosine distance to group benchmarks and visualize their relationships. Cosine similarity matrices and dendrograms illustrate these clusters.

2. Principal Components Analysis (PCA): We represent each benchmark as a feature array composed of scores from all models and project these arrays onto the two principal dimensions.

Figure 14 in the edited manuscript (Appendix A.15) presents the correlation matrix and dendrograms. Circles represent BigDocs-Bench tasks (our benchmarks), while triangles denote other document benchmarks included in the paper. These results demonstrate that BigDocs-Bench tasks are notably distinct from other benchmarks, as indicated by their low correlation with them. A clear grouping emerges for tasks related to VQA, as well as BigDocs-Bench tasks involving code generation in LaTeX, JSON, GraphViz, HTML, and SVG. Notably, HTML and SVG tasks form a cluster, likely due to their characteristically long output sequences. Additionally, DeepForm and KLC stand out as a separate cluster, distinct from all others.

Figure 15 in the manuscript illustrates the PCA results. The clusters distinctly separate BigDocs-Bench tasks from other benchmarks, reaffirming their unique characteristics. Notably, the average aggregated score from BigDocs-Bench (left side) is the furthest from the aggregated score of other benchmarks (right side), highlighting the distinctiveness of our benchmarks. However, certain tasks, such as Image2SVG and Screenshot2HTML, show stronger correlations with KLC, DeepForm, and TabFact. This is likely due to the shared level of difficulty across these benchmarks, which require nuanced understanding and complex reasoning.

Unique Aspects of BigDocs-Bench. Our correlation results show a clear separation between BigDocs-Bench and other document benchmarks. In summary, it introduces the following novel elements to the landscapes of VLM evaluation and document understanding:

1. Document and Chart Reasoning: Tasks focused on understanding, question answering, and summarization of charts, graphs, and workflows from images.
2. Web and GUI Understanding: Tasks targeting web and graphical user interface comprehension, a less explored area in current benchmarks.
3. Multimodal Code Generation: Tasks involving the conversion of images into code representations, such as inverse rendering and code generation (e.g., HTML, SVG, JSON).
4. Long-Context Structured Output Generation: Tasks requiring structured output over extended contexts, such as generating long JSON, SVG, HTML or Markdown from images