LMDX: Language Model-based Document Information Extraction and Localization

We would like to thank reviewer 5Y8u for taking the time to read our paper in detail and giving valuable feedback! Please see our response below:

The model's performance highly depends on the off-the-shelf OCR and PaLM2-S large language model, but they are unavailable, so the results are not reproducible. Also, there is no detailed explanation or evaluation of these modules.

[Response] We agree we did not make this clear in the submitted paper, but all our experiments use the OCR provided by the benchmark for all models and baselines. As such the reported performance improvements are entirely due to the model and LMDX methodology, and we can fairly compare it to the baselines. We will make this clearer in the next version of the paper.

The authors mention the support of the hierarchical entity and entity localization, but their effect is not directly evaluated since there is no evaluation without them.

[Response] With regards to hierarchical entity evaluation, we believe this is directly evaluated: we highlight the F1 score on line_item entity (the hierarchical entity in VRDU) for all models and baselines in Table 2 and a provide a discussion for it in Section 3.3 Performance on Hierarchical Entities. The table below synthetizes this information and shows the line_item F1:

Overall, previous SOTA methods (LayoutLM / FormNet) plateau around ~20-25% F1 due to their reliance on heuristics that do not benefit from additional training data, unlike LMDX. Please let us know if there are more that would be useful to show with regards to hierarchical entity evaluation.

We also acknowledge that in the submitted paper version, entity localization is not directly/independently evaluated, and we thank reviewer 5Y8U for pointing it out. For all models and baselines that can do entity localization, on VRDU Registration Form and Ad-Buy Form Mixed, we've computed the localization accuracy, following the formula below:

$Accuracy_{Localization} = TotalEntityCount_{CorrectlyExtractedAndLocalized} \div TotalEntityCount_{CorrectlyExtracted}$

Since LMDX localizes at the line level, localization verification is done at the line-level as well, i.e. localization is considered correct if the prediction bounding box is covered by the groundtruth line-level bounding box by more than 80%. See the results below.

VRDU Registration Form Mixed:

VRDU Ad-Buy Form Mixed:

This evaluates the localization quality independently of the extraction quality. We will update the next version of the paper to include those results.

Is there any performance assessment of the OCR model?

[Response] In all our experiments, we simply use the publicly provided benchmark's OCR for all models and baselines, thus we do not explicitly evaluate the performance of the OCR model. This means that the quality improvements showcased come from the parsing model (powered by LMDX, FormNet, LayoutLMv1/2/3, etc) and not the OCR itself, leading to a fair comparison between LMDX and baselines.

For details regarding the OCR, the VRDU authors mention they used Google Vision OCR (per page 5 of [1]).

[1] Wang, Zilong, Yichao Zhou, Wei Wei, Chen-Yu Lee, and Sandeep Tata. "Vrdu: A benchmark for visually-rich document understanding." In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5184-5193. 2023.

Firstly, we would like to thank reviewer W15R for taking the time to review our paper and providing great feedback! Please see our response below:

Though the suggested method seems agnostic to specific LLM, the authors experimented only with PaLM 2-s. To verify the superiority of the suggested framework, additional experiments using LLMs other than PaLM are needed (I think the additional experiment would enhance the presentation of the robustness of the proposed method).

[Response] We agree with the feedback that evaluation numbers with other LLMs would better show the generality of our method. What we meant by LMDX being generic to the LLM is that it simply requires a simple text-in, text-out interface, which all LLMs provide (no extra input/output, or reliance on specific tokenization scheme). We will rephrase this in the next version of our paper to make this clearer.

In document representation, when generating prompts, how well do coordinate tokens work? Line-level segments with 2 coordinates are enough for various VRD data?

[Response] Coordinate tokens are very important to a high quality extraction: this is showcased in the ablation study atTable 4, where we replace the coordinate tokens by line index (See appendix A.6 for how the prompts look). This allows us to not communicate any layout information to the model, while still using LMDX prompts and localizing the entities. Overall, coordinate tokens improves micro-F1 by an absolute 12-15% across data regimes on VRDU Ad-Buy Form Mixed.

Line-level segments with 2 coordinates are indeed enough based on our experiments. We’ve tried with 4 coordinates, word-level and finer coordinate quantization buckets on an internal dataset which were all quality-neutral or negatives. We posit that this is due to the fact that those schemes add more coordinate tokens to the text sequence, which makes it further and further from the type of data the LLM was pretrained on. It becomes then harder for the LLM to correctly interpret the LMDX-style text sequences.

Chunk 1, Sample 1: { “registrant_name”: “John 10|20” }
Chunk 1, Sample 2: { “registrant_name”: “John 10|20” }

Chunk 2, Sample 1: { “registrant_name”: null }
Chunk 2, Sample 2: { “registrant_name”: null }

Chunk 3, Sample 1: { “registrant_name”: null }
Chunk 3, Sample 2: { “registrant_name”: null }

We would like to thank reviewer Vt8a for taking the time to review the paper and give us insightful feedback! See our response below:

This paper proposes a mechanism for information extraction from documents leveraging off-the-shelf Optical Character Recognition service and complicated LLM prompt engineering approach for processing the extracted information. The main underlying assumption driving the complexity of the prompt engineering approach is limited context length of LLMs. However, models like Claude 2 are capable of working with 100K token context windows. Additionally, methods like RoPE scaling and other context length expansion approaches allow to increase the context size for other LLMs including open-source models. As there are effective ways to address the context length limitation, the presented prompt engineering approach is a somewhat incremental engineering contribution, especially given its complexity. The fine-tuned model, however, is of interest.

[Response] To clarify, the complexity of the prompt engineering is not driven by a goal to reduce the number of tokens due to limited context length, but rather to communicate the layout information of the document to the model (necessary for a high-quality extraction on many visually-rich documents), and to derive the extracted entities’ bounding boxes (a major requirement in document information extraction which we want our method to handle). Both are achieved through the coordinate tokens contained in the prompt.

The chunking stage in the pipeline (Section 2.3), which reduces the number of tokens sent in each prompt, is detailed so that ML practitioners can use the methodology on LLMs with context length limitations or on LLMs that can not use their full context well [1]. However, that stage can be entirely skipped for LLMs that supports long context and can use it effectively. We will ensure to clarify this in the next version of our paper.

Nonetheless, we do make an effort to reduce the number of tokens used by our method whenever it is quality neutral so that it can be used by ML practitioners at a lower cost and faster inference speed, but this is not the primary goal.

Overall, we hope this clarifies details around the complexity of the prompt!

[1] Liu, Nelson F., Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. "Lost in the middle: How language models use long contexts." arXiv preprint arXiv:2307.03172 (2023).

{RAW_OCR_TEXT}

Given the document, extract the text value of the entities included in the schema in json format.
- The extraction must respect the JSON schema.
- Only extract entities specified in the schema. Do not skip any entity types.
- The values must only include text found in the document.
- Use null or [] for missing entity types.
- Do not indent the json you produce.
- Examples of valid string value format: "$ 1234.50", "John Do", null.
- Examples of valid list value format: ["$ 1234.50", "John Do"], [].

Schema: {"file_date": "", "foreign_principle_name": "", "registrant_name": "", "registration_num": "", "signer_name": "", "signer_title": ""}
```json

<DOCUMENT_IMAGE>

Given the document, extract the text value of the entities included in the schema in json format.
- The extraction must respect the JSON schema.
- Only extract entities specified in the schema. Do not skip any entity types.
- The values must only include text found in the document.
- Use null or [] for missing entity types.
- Do not indent the json you produce.
- Examples of valid string value format: "$ 1234.50", "John Do", null.
- Examples of valid list value format: ["$ 1234.50", "John Do"], [].

Schema: {"file_date": "", "foreign_principle_name": "", "registrant_name": "", "registration_num": "", "signer_name": "", "signer_title": ""}
```json

Changes made to the paper

Following the received feedback, we've done the following changes to the paper:

• (reviewer 5Y8u) We've added an evaluation and discussion of the localization accuracy in Table 2 and Section 3.3 (page 7-8). This allows to compare LMDX's localization performance versus baselines, independently of the extraction quality.
• (reviewer 5Y8u) We've clarified that Line Item F1 in Table 2 and Section 3.3 (page 7) is the evaluation of hierarchical entity performance, and independently compares how LMDX F1 performance on hierarchical entities compares to baselines.
• (reviewer Vt8a) We've added large models baselines to the results Tables 2 and 3 (section 3.3, page 7-8). Those baselines are using GPT-3.5 and LLaVA-v1.5-13B and are fully detailed in Appendix A.7. Unlike LMDX, those baselines do not allow localizing the entities, and as such do not meet all desired properties of document information extraction systems.
• (reviewer Vt8a) We've added section 3.5 (page 8-9) which contains in-context learning results on CORD with multiple retrieval methods, and discussion of those results. Overall, nearest neighbors work the best, as it retrieves documents from the same template as the target document (see Appendix A.10 for a more detailed discussion and sample retrievals).
• (reviewer W15R) We've added discussion on sampling error rates in Appendix A.9, detailing the different errors that can happen during decoding of the LLM's completions. Errors are very low overall, and with our sampling strategy, all documents on our benchmarks have a successful extraction.
• (reviewer Vt8a) We've clarified the role of chunking in section 2.3 (page 3) with regards to long-context LLMs.
• (reviewer 5Y8u) We've clarified in section 3.2 (page 6) that we use the benchmark's provided OCRs for all our experiments and for all models and baselines taking text as input, leading to a fair comparison between them.
• (reviewer W15R) We've clarified in the contributions in Section 1 (page 2) that LMDX only requires a simple text-input/text-output interface that all LLMs provide, and does not rely on other details such as a particular tokenization.
• (reviewer Vt8a) We've defined the unexplained abbreviations (IOB, NER) before using them.
• Finally, we've done general rephrasing and small changes to make everything fit within the page limit. In particular, Figure 3 (Diagram about the LMDX training phases) was removed as it did not provide any additional information from what was contained in the text content.

We would like to thank you for taking the time to provide insights and suggestions on our paper. We do believe it improved our paper significantly! Please let us know if any question/concern remains unanswered, and happy Thanksgiving if you're celebrating it!