A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding

Thank you for taking the time to review our paper. We sincerely appreciate the reviewer’s recognition of our work, the thoughtful feedback, and the opportunity to address the concerns raised.

W1-Plora limites the novelty: The core contribution of this work lies in the SLP module, which effectively unifies layout and text, along with the tailored LNTP and SSFT training tasks, both of which were well-received by reviewers. In contrast, the impact of P-LoRA on performance improvement is relatively minor. As shown in Table 4, comparing the second and third rows reveals that adding P-LoRA increases the average accuracy by only 0.8% for KIE and VQA tasks. This improvement is significantly smaller compared to the introduction of SLP, as evident when comparing the first and second rows in Table 4.

W2-analysis of SLP is incomplete:

• We will add a new experiment regarding the analysis of impact of SLP in model's language ability, more details see response for Q3.
• We will provide a clearer description of the ablation studies in Section 4.6 to emphasize the effectiveness of SLP. Specifically, the first row of Table 4 represents the standard Llama2 model without SLP. This setup uses the same training data as the second row, including datasets such as DDD, Layout-aware SFT, DocVQA, InfoVQA, ChartQA, FUNSD, CORD, SROIE and VisualMRC. Both training is normal SFT (No LNTP and No SSFT). The key distinction between the two rows lies in their inputs: the second row integrates layout bounding box information using SLP, whereas the first does not. This comparison underscores the impact of SLP on enhancing document understanding performance.

Q1 token statisitcs: The token statistics for LNTP and SSFT using the Llama2 tokenizer are summarized below and will be included in the updated version.

Q2 why P-Lora works: As noted in response to W1, while P-LoRA is effective, its impact is not particularly significant. Actually, in our experiments conducted on an in-house dataset, we observed that when the training set is sufficiently large, P-LoRA becomes dispensable ss shown below. We believe that the effectiveness of parameter-efficient fine-tuning methods—such as LoRA, P-LoRA, prompt tuning, and P-tuning—diminishes as the training set size increases. In such cases, fully supervised fine-tuning emerges as the most effective approach.

Q3 impact on language ability after incooporating SLP Good question. We conducted an additional experiment to evaluate the language understanding ability of the models after integrating spatial information. Specifically, we measured the perplexity on the Wikitext test set for baseline and LayTextLLM models. Perplexity serves as an indicator of language understanding, with higher values suggesting a potential decline in language ability compared to the baseline (LLaMA2-7B). The results, as shown in the table.

• LayTextLLM_zero: it shows low perplexity, indicating spatial information can be integrated with a little affecting language modeling.
• LayTextLLM_vqa: Highest perplexity, maybe due to DocVQA and InfoVQA datasets, emphasizing document reasoning, which shifts focus from general language to document-specific tasks.
• LayTextLLM_all: very similar perplexity t LayTextLLM_zero and baseline, again demonstrate SLP has little impact on language ability.

We will add this experiment in the updated version.

We kindly request your acknowledgement of our reply. We would fully appreciate it if you would consider improving the rating. We look forward to your response.

Thanks for reviewing our paper. We appreciate the reviewer’s acknowledgment of our contributions, including the SLP module, input length reduction, and performance improvements. We are grateful for the thoughtful feedback and take the opportunity to address the concerns raised.

Paradigm is quite similar: We admit our approach follows the standard pretraining and fine-tuning paradigm; however, we introduce innovative modifications to the model architecture, i.e. SLP, enabling us to design tailored pretraining (LNTP) and fine-tuning tasks (SSFT). These novel tasks, appreciated by reviewers mYNU and p4cT, are recognized for their innovation and enhancing the generalizability of LLMs.

Compared with more recent Text+ methods: According to ICLR guidelines, authors need not compare with papers published after July 1, 2024, or non-peer-reviewed works like those on arXiv. However, we prioritize a thorough evaluation and will include the most recent relevant Text+ methods. While we found no new Text+ methods, we identified the Visual+Text+Coordinate work DocLayLLM [1] and will add it to Table 3 in the updated version.

[1] Liao, W., Wang, J., Li, H., Wang, C., Huang, J. and Jin, L., 2024. DocLayLLM: An Efficient and Effective Multi-modal Extension of Large Language Models for Text-rich Document Understanding. arXiv preprint arXiv:2408.15045.

Error bold font in Table 1: Thank you for highlighting the error in bold font; we will correct it in the next version.

We kindly request your acknowledgement of our reply, and are welcome to further discussions for your questions and concerns. We would fully appreciate it if you would consider improving the rating. We look forward to your response.

Thank you for taking time to review our paper. We thank the reviewer for the thoughtful feedback and are grateful for the opportunity to address the concerns raised.

W1-the error accumulation and latency reduction due to OCR engine:

• OCR engine error accumulation: We acknowledge the potential for error accumulation in OCR engines. However, MLLMs currently excel in reasoning over cognition, as shown by lower KIE task performance in Table 1 and the the new results compare to most recent SOTA MLLM below. We observed many errors of MLLM stem from the limited text recognition capabilities of current MLLMs, resulting in inaccuracies such as misreading "169.80" as "169.78." While our method outperform in KIE tasks by a large margin, again show the efficacy of OCR-based method in VRDU.

• OCR throughput: Off-the-shelf OCR modules use lightweight text detection and recognition models with tens of millions of parameters, far smaller than MLLMs or visual encoders like ViT (400M parameters). Our model uses only bounding box and text tokens (without visual tokens), making OCR integration negligible for real-world inference latency.

W2-performance drop compared to LayoutLLM and in ChartQA:

• Lower accuracy compared to LayoutLLM: LayoutLLM outperforms due to incorporating visual input alongside OCR text and bounding boxes but at higher computational cost (visual token + OCR texts token). Our approach seeks a lightweight alternative, reducing sequence length by focusing on efficient representations only involving OCR texts and bounding box, aiming to surpass other OCR-based methods (Table 2).

• Inferror performance in ChartQA: there are two reasons:

  • (1) In Appendix E, we note that LayTextLLM, using only text and layout inputs, struggles with visual reasoning tasks like ChartQA, which require cues such as length, size, color, and area (Figure 7). As a result, it underperforms compared to MLLMs with visual inputs. Previous OCR-based methods often omit ChartQA results due to its challenging characteristics.

  • (2) We use exact match accuracy for consistency across datasets, though ChartQA's standard metric is relaxed accuracy (within 5% tolerance). Re-evaluated results (relaxed accuracy) are shown below. While LayTextLLM remains less competitive than some OCR-free methods due to ChartQA's visual challenges, it achieves results comparable to MLLMs (e.g., LayTextLLM_all: 43.2 vs. InternVL: 45.6). These updates will appear in the revised version. | Model | Relaxed Accuracy | Exact Match Accuracy | |------------------|------------------|-----------------------| | LayTextLLM_zero | 28.1 | 22.8 | | LayTextLLM_vqa | 30.1 | 23.4 | | LayTextLLM_all | 43.2 | 35.4 |

W3-ablations are unclear: The first row in Table 4 represents the standard Llama2 model without incorporating any layout information, taking only OCR text as input. This setup uses the same training data as the second row, which includes all VQA and KIE datasets. The key difference between the two rows lies in the model inputs: the second row integrates layout bounding box information via SLP, while the first does not. Both training is normal supervised fine-tuning (No LNTP and No SSFT).The comparison betwen first two rows highlights the effectiveness of SLP. Notably, we excluded normalized [x1, y1, x2, y2] coordinates in the first row because their inclusion significantly degraded performance during our preliminary experiments. This decline may result from the excessively long sequences and the requirement for a larger-scale LLM, as discussed in lines 448–449. We will add a clearer description of ablation study in the updated version.

Q1-report number error: Thank you for pointing this out. We chose to present the LayoutLLM_CoT (Vicuna-7B) because it has a higher score. However, we agree that a fairer comparison is necessary. Therefore, we will update the result using Llama2-7B as shown below and this will be reflected in the updated version (the original results are in brackets and are obviously higher will not show in the updated version).

Q2-subperformance in ChartQA: refer to Appendix E and our response to Weakness 2.

We kindly request your acknowledgement of our reply. We would fully appreciate it if you would consider improving the rating. We look forward to your response.

Thank you for taking time to review our paper. We thank the reviewer for the thoughtful feedback and are grateful for the opportunity to address the concerns raised.

First of all, our contributions are not trick, which were well-received, with reviewers (mYNU, ZrGX, and p4cT) praising the introduction of SLP for unifying layout and text, and acknowledging improved performance. LNTP and SSFT tasks were highlighted as innovative, and the input length reduction further validated our approach.

Research Problem Validity:

• without incorporating visual features: the reviewer has misunderstanding with the paper. The work acknowledges the importance of incorporating visual features (line 906-908) and recognize it as limitation as discussed in Appendix E. The key of the works is taking a different approach by focusing on spatial information and textual data rather than full visual embeddings. OCR-based methods are useful as a practical and computationally efficient solution​. Experimental results demonstrate significant improvements over MLLMs in tasks such as KIE (27.2% improvement) and our newly added result comparing with the most recent SOTA MLLMs InternVL2. These results suggest that spatial-textual integration can achieve competitive performance without directly relying on raw visual features.

• MMMU-PRO findings:

  • The findings are based on the MMMU-PRO is a benchmark that intentionally filters out questions answerable directly from text (Section 2.2 of MMMU-PRO paper), which explains the similar performance with and without OCR. Their experiment is actually designed to emphasize the challenging nature of their benchmark (focusing on questions that cannot be answered without visual input) rather than to prove that text-only approaches are insufficient for document understanding. In real-world applications, many VRDU tasks can be effectively addressed using text and layout alone. Thus, the conclusions drawn from MMMU-PRO are not entirely applicable to the scope of our work, which focuses on leveraging text and layout information efficiently for practical document understanding scenarios.

  • The findings from MMMU-PRO are conditioned by using large 70B open-source models or other competitive proprietary models. However, when OCR is combined with smaller models, significant performance improvements are observed. This underscores the value of LayTextLLM, which provides an efficient solution that leverages OCR and spatial-text integration to achieve strong results, even without relying on massive model sizes.

Scope and Contribution:

• Contribution: contribution is substantial and well-received by the reviewers. Reviewer mYNU, ZrGX, and p4cT have praised the introduction of SLP, which unifies layout information and text into a cohesive sequence representation. All reviewers acknowledged the improved performance achieved by our approach. Specifically, the tailored tasks LNTP and SSFT were highlighted as innovative and effective by Reviewers mYNU and p4cT, while Reviewer ZrGX appreciated the reduction in input length, further validating the impact of our contributions.

• Scope: The scope of this work is broad and significant for the VRDU field, encompassing key tasks like VQA and KIE. Prominent datasets such as DocVQA, FUNSD, and SROIE, widely used to evaluate both previous high-profiling models (e.g., LayoutLM series) and most recent text-centric MLLMs/LLMs (e.g., InternVL, Qwen2 VL, TextMonkey), highlight its relevance. Our method achieve improved performance (appreciated by all reviewers) on all these datasets, again demonstrate the contribution of the work.

• exclusion vision: As clarified in the response to Research Problem Validity, the work recognizes the importance of visual features in document understanding but adopts a different approach, focusing on spatial and textual information instead of full visual embeddings. The importance of visual information is acknowledged, and we have noted plans to explore its integration in future work (see Appendix E, lines 906–908).

Broader Impact and Audience: Our submission aligns with the ICLR track (our primary area for the submission is:applications to computer vision, audio, language, and other modalities). Our work specifically unifies layout tokens and text tokens, significantly enhancing the application of LLMs in document understanding, making it well-suited for ICLR's audience.

We kindly request your acknowledgement of our reply, and are welcome to further discussions for your questions and concerns. We would fully appreciate it if you would consider improving the rating. We look forward to your response.

Contribution: Our contribution is a novel method to integrate layout information into LLMs by interleaving layout tokens with text tokens, unifying all inputs into a single sequence. We also propose two tailored training tasks that significantly enhance the performance of OCR-based approaches in VRDU tasks. Our method achieves substantial improvements over existing OCR-based state-of-the-art approaches, such as DocLLM and ICL-D3IE, demonstrating a large performance margin (Table 2). All contributions are well-received and recognized by other reviewers.

Experimental Setup: Our experimental setup ensures fairness. In Tables 1, 2, and 3, datasets used in the SFT are marked with an asterisk (*). The improvements mentioned in Abstract, line 24, are based on the "zero-shot" setting in KIE (Table 1: LayTextLLM_zero 67.6 vs. TextMonkey+ 40.4) and the SFT setting in VQA (Table 1: LayTextLLM_VQA 59.7 vs. TextMonkey+ 47.7), as all VQA data is utilized in the SFT for these MLLMs.

During the submission, InternVL2-8B is excluded from the comparison to comply with the ICLR complementary policy.. As you mentioned, "datasets like FUNSD and SROIE were indeed included in the VLM training process, albeit as part of a broader training set," which indicates that models like InternVL2 benefit from pretraining on these benchmark datasets (though not through SFT). In contrast, LayTextLLM does not （i.e., LaytextLLM_zero）, further highlighting the strong performance of the proposed method.