A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding

Thank you for your thoughtful feedback and appreciate the recognition of our paper’s contributions and novelty. We are grateful for the opportunity to address the concerns raised.

W1-Model backbone: We implemented LayTexLLM using Llama2-7b, consistent with previous OCR-based methods like DocLLM, which also use Llama2-7b. We also replicated the results of the coor-as-tokens scheme using Llama2-7b for consistency. Noting the LayoutLLM model utilizes Llama2-7b and Vicuna 1.5 7B, which is fine-tuned from Llama2-7b. Therefore, for the majority of our comparisons, the models are based on the same or similar LLM backbones, allowing for a fair comparison between approaches.

Other MLLMs use backbones like Qwen-VL, Internlm, and Vicuna, which are models with at least 7 billion parameters, excluding the visual encoder. Thus, we can say the comparison is fair, at least in terms of model parameters. We will explicitly mention this in the updated version.

W2-OCR engine: We use word-level OCR from the respective datasets to ensure a fair comparison, except for the ChartQA dataset, where no OCR is provided (and we have mentioned this in lines 277-278). We will explicitly mention this in an updated version.

W3-Confusion of terminology:

Sorry for the confusion caused. Here are explanations:

• Pretrain and SFT Clarification: In Section 4.1, the terms "pretrain" and "SFT" refer to LNTP and SSFT. We will revise this section to avoid confusion.
• Zero-shot and Supervised Results: The term "zero-shot" refers to a model trained using SSFT only with Document Dense Description (DDD) and Layout-aware SFT data, as used in LayoutLLM. "Supervised" indicates that the model is trained using SSFT with DDD, Layout-aware SFT data, and the training sets of downstream datasets such as DocVQA and FUNSD. This terminology aligns with LayoutLLM, and we will clarify this in the updated version.
• Asterisk Notation: An asterisk (*) is used to indicate whether the corresponding training set of a downstream dataset is included in the training of a specific model. This notation facilitates a fair interpretation of experimental results for the reader.

W4-Statement about improvement: We'll tone down the phrasing to accurately reflect this comparison and highlight our improvements in relation to OCR-based methods.

W5-LNTP: We acknowledge that LNTP resembles the regular language modeling objective. We'll tone down the presentation to clarify that the added SLP and P-LoRA parameters are updated using standard language modeling loss.

Q1-claim of autoregressive vs. docllm: We had a brief discussion in line 297 – line 299. Here we elaborated this in detail, and add them in the updated version:

• Disentangled Attention: DocLLM uses a disentangled attention to process spatial and textual modalities separately (using differerent QK weight) before integrating them. This independent handles of spatial information from document layouts, unlike traditional autoregressive models that process inputs sequentially (use the same suite of weights). In contrast, LayTextLLM interleaves bounding box tokens with text, unifying both modalities in a single sequence through an autoregressive approach.

• Block Infilling Objective: Unlike standard autoregressive models that predict the next token based only on preceding text, DocLLM uses block infilling to predict missing text blocks based on both preceding and succeeding context. This deviates from leveraging the inherent autoregressive nature of traditional LLMs which solely relies on preceding tokens.

• Impact on Performance: As demonstrated in Table 2. when compared using the same training dataset, LayTextLLM significantly outperforms DocLLM.

Q2-encode x and y: We considered using positional encodings that directly encode (x, y) positions to address the artifact issue. However, to fully leverage the LLM parameters and maintain simplicity, we avoided encoding (x, y) positions, as it could complicate the model. Instead, we focused on balancing LLM reuse with necessary adjustments, which led us to propose SSFT.

Q3-model backbone and generalization of the method: We implemented LayTexLLM using Llama2-7b as our LLM backbone, in line with prior OCR-based methods like DocLLM and LayoutLLM.

Our method is model-agnostic. In our in-house KIE test, we evaluated the performance of the Baichuan2 7b and Qwen2 7b models. The results showed that incorporating the SLP layer improved performance for both models compared to not using it.

Q4-using PLORA: The motivation for using P-LoRA is due to the concern of having too few learnable parameters. P-LoRA is applied in each layer, but the main contribution still comes from SLP.

Q5-LNTP loss: Yes, the bounding box tokens are used as inputs but not targets. Our objective is to understand the bounding boxes, not to generate them. Therefore, it is unnecessary to compute a loss for the bounding box tokens.

Also, we tested in a in-house KIE dataset, finding that including bounding box as targets (using string like ''[1,20,10,30]'') during LNTP drops the downstream performance.

Q6-typo: We will fix those typos in the updated version.

Q7-polys and asterisk: The term 'polys' will be replaced with 'coordinates.' An asterisk (*) indicates if the training set of a downstream dataset is included in the training of a specific model. This notation ensures a fair interpretation of the experimental results for the reader.

Limitations: Yes, bolding and italic texts should be included as visual cues, which will be updated in the updated version.

We would appreciate it if you could improve your rating if all concerns been addressed and we look forward to your response.

We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s contributions, writing and novelty. We are grateful for the opportunity to address the concerns raised.

W1-Compute loss of bounding box:

• First of all, our objective is to understand the bounding boxes, not to generate them. Therefore, it is unnecessary to compute a loss for the bounding box tokens. Also, as shown in Figure 4, the prediction of t2 is made using the hidden state of b1, which means the supervised signal is backpropagated to the SLP.

• We tested including bounding boxes as targets (using strings like "[1,20,10,30]") during LNTP on an in-house KIE dataset and found that performance dropped.

• However, when we tested including bounding boxes as targets during SSFT, we found that the precision increased while the recall decreased, resulting in an almost unchanged micro F-score. Therefore, we can conclude that including bounding boxes as targets is beneficial only when added in the downstream tasks instead of during pretraining, and only when the application is sensitive to precision.

W2-Including visual modality: We acknowledge that incorporating visual information can enhance performance, as discussed in the Limitations section. Exploring this further is a direction for our future research.

Q1-length reduction: The length reduction is universal and can be generalized to other LLMs. We have conducted additional tests on sequence lengths using the Baichuan2 tokenizer on an in-house KIE dataset, confirming the token reduction is universal and agnostic to LLMs used when compared to coor-as-tokens. When compared to DocLLM, we can ensure that LayTextLLM maintains an equal or shorter sequence length, regardless of the tokenizer used.

Thank you very much for your valuable feedback and keeping the positive rating. We are committed to incorporating all of the clarifications you suggested, particularly regarding the discussion of bounding box prediction in the final version of our paper.

We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s novelty and improved performance. We are grateful for the opportunity to address the concerns raised.

W1-Lack visual modality: We acknowledge that incorporating visual information can enhance performance, as discussed in the Limitations section. Exploring this further is a direction for our future research.

W2-Exploration of shuffle ratio:

We conducted an additional experiment using DocVQA, and the results again demonstrated the superiority of LayTextLLM, and confirming that choosing 20% is an appropriate selection. We will add this result in the updated version.

W3-More ablation studies:

We have conducted a new ablation study, which will be added in the next version. The experimental results demonstrating the interleaving of bounding box and text providing the largest boost, while LNTP+SSFT provide big improvement in VQA tasks.

Note the slight difference in the value from the previous version is due to our use of an in-house framework, while the new version is based on the Huggingface Transformers.

W4-fair comparison

• We use an asterisk (*) indicates whether the training set of a downstream dataset is included in the training of a specific model, which ensures a fair interpretation of the experimental results for readers. While we acknowledge that LayoutLLM performs better in the zero-shot DocVQA scenario, our primary comparison focuses on pure OCR layout + OCR text models, such as DocLLM and ICL-D3IE, as LayoutLLM incorporates the visual modality.
• Could you please specify the citation that provides the zero-shot performance of LayoutLMv3, and we will include this information in the updated version.

Q1-advantage of projecting bounding box

• Precise Layout Representation: The LayoutLM series uses position embeddings for discrete layout representation, while LayTextLLM maps four coordinates into a continuous hidden space which is continuous. We believe this approach offers a more precise and enriched understanding of layout.
• Enhanced Contextual Understanding: By interleaving spatial layout with textual content, the model enhances its understanding of context and structural relationships within documents. This is especially beneficial for layout-dependent documents such as invoices, forms, and multi-column scientific articles, and is particularly advantageous for decoder-only models like LLMs.

We kindly request your acknowledgement of our reply, and are welcome to further discussions for your questions and concerns. We would be fully appreciated if you would consider to improve the rating. We look forward to your response.

Thank you for taking time to review our paper. We thank the reviewer for the thoughtful feedback and are grateful for the opportunity to address the concerns raised.

W1-Incomplete related work: Our survey focuses on decoder-only architectures within LLMs to highlight their unique capabilities. We acknowledge the need for more comprehensive coverage and will include citations on encoder-only architectures in the updated version.

W2-Insufficient explanation:We had a brief discussion in line 297 – line 299. Here we elaborated this in detail, and add them in the updated version:

• Disentangled Attention Mechanism: DocLLM introduces a disentangled attention mechanism that processes spatial and textual modalities separately. This mechanism handles spatial information independently before merging it (using different attention QK weight) with textual information, which is different from traditional autoregressive models that process inputs sequentially without such separation (using the same suite of weights). While LayTextLLM introducing bounding box token interleaving with texts, unifying bounding box and text information in a single sequence, fully fusing these two modalities by autoregressive method.

• Block Infilling Objective: Unlike standard autoregressive models that predict the next token based on the preceding sequence, DocLLM uses a block infilling approach where it predicts missing text blocks based on both preceding and succeeding context. This deviates from leveraging the inherent autoregressive nature of traditional LLMs which solely relies on preceding tokens.

• Impact on Model's Predictive Performance: The experimental findings highlight the superiority of our method, LayTextLLM, as demonstrated in Table 2. When compared using the same training dataset, LayTextLLM significantly outperforms DocLLM.

W3-Limited comparison

• ICL-D3IE: We have included a comparison with ICL-D3IE. The data in Figure 1 and Table 2 is sourced from the ICL-D3IE paper (Coord-as-tokens-ICL-175B(Davinci-003), Table 2 Davinci-003-175Bcoor). We also replicated the ICL-D3IE using LLama2-7b (Figure 1, Coord-as-tokens-7B(Llama2), Table 2 Llama2-7B-chatcoor). A detailed discussion of the comparison with ICL-D3IE can be found in lines 301-307. In the updated version, we will change the term "coor-as-tokens" to "ICL-D3IE."
• Lmdx: Our comparison primarily focuses on approaches based on open-source LLMs instead of proprietary ones, as noted by Reviewer teB5. This ensures our results are reproducible. Therefore, LMDX was not included. Additionally, LMDX only provides results for CORD, whereas our experiments cover a broader range of text-rich VQA and KIE datasets. However, we will include a comparison with LMDX in the updated version.
• Shikra: Shikra is not a document AI LLM even without proper OCR ability and is therefore outside the scope of this comparison. As noted by Reviewer teB5, the primary comparison should be with other OCR-based methods, such as DocLLM.

W4-Marginal token reduction

• We utilize word-level OCR from the corresponding datasets to ensure a fair comparison, which will be explicitly mentioned in a later version of the document.

• Our claim of significant token reduction is primarily focused on the comparison with the coor-as-tokens scheme, as detailed in lines 74-77. In these instances, the reduction in tokens is substantial rather than marginal and applies to both word-level and line-level OCR. For example, when using the coord-as-tokens scheme with the Llama2 tokenizer, the coordinate string "[70,73,90,77]" occupies 13 tokens, while LayTextLLM represents the same information with just 1 token.

• Furthermore, compared to DocLLM, our approach yields either shorter or equivalent sequence lengths. We have conducted additional tests on sequence lengths using the Baichuan2 tokenizer on an in-house KIE dataset, confirming the token reduction is universal and agnostic to LLMs used.

Q1-encoding vector: Using an embedding to encode coordinates is not about introducing additional information. Instead, it is a practice about transforming the coordinates into a hidden state that is more understandable for a LLM. This process involves aligning dimensions in a way that is more suitable for the model's architecture. For example, in LLAVA, the 1024-dimensional output from CLIP is mapped to 4096 dimensions to better align visual and text modality.

Q2-Train SLP: There seems to be a misunderstanding regarding the training process. Although the loss is not computed for the bounding box token, the SLP is still trained. For instance, in Figure 4, the prediction of t2 is made using the hidden state of b1, which means the supervised signal is backpropagated to the SLP. Furthermore, our objective is to understand the bounding boxes, not to generate them. Thus, it is unnecessary to compute a loss for the bounding box tokens.

Q3-performance of LaytextLLM_zero: The strong performance is primarily due to the interleaving of bounding box tokens with text, along with other design elements such as LNTP and SSFT. Additionally, the use of synthetic data from LayoutLLM, including DDD and layout-aware SFT, also contributes to training LayTextLLM_zero.

We kindly request your acknowledgement of our reply, and are welcome to further discussions for your questions and concerns. We would be fully appreciated if you would consider to improve the rating. We look forward to your response.