PPTSER: A Plug-and-Play Tag-guided Method for Few-shot Semantic Entity Recognition on Visually-rich Documents

Thank you for your patient review. Our following responses will address your concerns.

Weakness

• Weakness #1: Novelty Concerns

  • We appreciate the observation regarding the application of cross-attention in NER for pure text. However, our approach to implementing cross-attention for few-shot SER is unique; it leverages the intrinsic self-attention mechanism within a pre-trained Transformer model, as detailed in Section 3.2.2. Unlike other methods that employ an additional cross-attention module, our design relies solely on the core self-attention operation of the transformer architecture, distinguishing it from conventional methods. Different from many other approaches that use the hidden state derived from cross-attention for classification-like operations, we directly use the attention weights between document tokens and tag-related prompts as the probability distribution of tokens belonging to distinct SER tags. This approach eliminates the conventional operation of multiplying attention weights with the value layer and then feeding the output to a feed-forward layer, saving parameters and computational time, as described in contributions in Section 1 and the subsection of Parameter Efficiency in Section 5.
  • Furthermore, we have also innovatively utilized the multi-head attention design of the attention mechanism. By selecting the maximum value from different attention heads, we ensure that each head focuses on different aspects. This also allows for a mutual backup function between different attention heads, which is particularly suitable for the entity-rich scenario on visually-rich documents. The effectiveness of this design is validated in Aggregation of Attention Weights of Section 5.
  • Importantly, methods used for NER on plain text may not necessarily be applicable to visually-rich documents. As shown in Table 2, we compared some modified NER methods for pure text with our method in the context of SER on visually-rich documents. The results clearly show that our method outperforms the modified pure text methods, underlining a clear distinction between SER on visually-rich documents and NER on pure text and reinforcing the significance of our research in this field.
  • Furthermore, multi-modal pre-trained models, typically pre-trained on structured documents, differ from the corpora used to pre-train BERT, which is closer to natural language expressions. Hence, exploring whether these multi-modal pre-trained models can understand the semantics of SER tags, which is closer to natural language expressions, and use them to guide the SER task is a valuable question. Our approach has proven effective across various multi-modal pre-trained models.
  • Also, our method could effectively type the entities and identify the boundaries between them. We split SER on visually-rich documents into two distinct tasks: entity typing and span detection. Innovatively, we explored the feasibility of using natural language descriptions as prompts (beginning and inner) for span detection, allowing us to simultaneously handle both tasks within a single framework.

• Weakness #2: Questions on label leakages

  We appreciate your concern, but it seems there may be a slight misunderstanding regarding our experimental setup. As mentioned in Section 3.1 of our paper, we employ an In-Label-Space setting where the label space for both the training set and test set is identical. Hence, they use the same tag-related prompt. However, this does not imply a label leakage issue in our study. Our research aims to improve the model's ability to efficiently extract information from novel types of visually-rich documents, such as invoices, receipts, etc., with minimal data annotation. It is not focused on extracting new entity types from existing visually-rich documents. The challenge is to design the model to handle a novel document scenario with limited labelled data, not to introduce novel entity types. Therefore, the use of the same prompts in testing does not lead to label leakage but rather aligns with our experimental design and objectives.

• Weakness #3 and Question #2: Questions on the representation of figures

  We are grateful for your suggestion. Figure 1 was designed to provide an intuitive comparison between our PPTSER method and the conventional fine-tuning approach in terms of parameters and structure. However, we recognize that this could lead to a potential content overlap between Figure 1(b) and Figure 2.

  In response to your comment, we have made preliminary modifications to the PDF version of our paper and are still actively seeking a more effective way to present this information. Rest assured, we will revise and update our paper accordingly in the final version to ensure clarity and ease of understanding.

Thank you for your patient review. Our following responses will address your concerns.

Weakness

• Weakness #1: Questions on the representation of figures and the formulations

  Thank you for your suggestion. Our original intent was to provide an intuitive understanding of our model's design through Figure 2, but we recognize that the representation might have inadvertently introduced some complexities. To address your concern, we have made modifications to Figure 2, removing certain non-essential elements to improve its clarity and intuitiveness. We will also polish the presentation to achieve a more formal and streamlined formulation of our model without sacrificing the necessary details of our neural architecture.

  These updated figures and the refined formulation of our method will be included in the final versions of our paper. We hope these minor adjustments will enhance the readability and comprehension of our work.

• Weakness #2, Weakness #3, and Question #3: Implementation details

  Thank you for your suggestions. Due to space constraints in the main text, we have included additional experimental details in Appendix A.2, and provided examples to illustrate how our PPTSER is trained in Appendix A.3 in our updated PDF files.

Question

• Question #1: Questions on labels for span detection

  We apologize for the confusion caused. As you correctly pointed out, when the label for span detection ${y}^{det.}_i$ is set to -1, it implies that we ignore the loss for span detection at this position. We have already made this correction in the main text.

• Question #2: Questions on how ${Q}$ and ${K}$ are calculated

  We apologize for not explaining this clearly in the main text. ${Q}$ and ${K}$ correspond to Query and Key in the attention mechanism, respectively. When the hidden state from the second last layer ${H}^{t-1}$ is segmented into ${H}^{t-1}_i$ along the channel dimension, it is multiplied separately with learnable weight $({W}^{t}_i)_q$ and $({W}^{t}_i)_k$ to yield queries ${Q}^{t}_i$ as well as keys ${K}^{t}_i$ as follows:

  ${Q}^t_i = ({W}^{t}_i)_q \times {H}^{t-1}_i$

  ${K}^t_i = ({W}^{t}_i)_k \times {H}^{t-1}_i$

• Question #4: Replacement proportion of unrelated words and random embeddings

  In Section 5, under the subsection of Prompt Engineering, when we mention unrelated words or random embeddings, we refer to replacing all tag names in the prompt with unrelated words or random embeddings. For instance, replacing menu.unitprice in the prompt with apple, or replacing its embedding with a randomly initialized tensor, which can be considered as a 100% replacement rate. This approach tests whether the semantic information contained in the tag names truly aids in completing the SER task. We apologize for any confusion caused by the text in Section 5. We will take steps to make this point clearer in the final version of the paper.

Once again, we would like to express our gratitude for your diligent review and valuable feedback during the peer-review process.

Weakness

• Weakness #1: Novelty Concerns

  I appreciate your feedback on the novelty of our work. While we agree that using NER tags as prompts has been explored in text-based NER, the novelty of our work lies in the application to Semantic Entity Recognition (SER) tasks in visually-rich documents, a domain that has not been thoroughly studied yet. The following points summarize our contributions:

  • Exploration of SER Tags' Effect on Visually-Rich Documents: Unlike pure text pre-trained models such as BERT, the corpora used for pre-training multi-modal pre-trained models do not mimic natural language expression. Considering the fact that multi-modal pre-trained models are trained on structured documents whose corpora present a gap between the natural language expressions. This presents a new question of whether SER tags, which are closer to natural language expressions, can be understood by multi-modal pre-trained models and guide SER tasks. Our experiments have successfully validated the effectiveness of the proposed approach.

  • Adaptation for Visually-Rich Documents: We have shown that text-based few-shot NER methods may not be as effective for the SER task on visually-rich documents. However, Our method, which is adapted for entity-rich scenarios on visually-rich documents, outperforms these conventional methods, highlighting the significance of this research.

  • Efficient Use of Multi-Head Attention Mechanism: Our method extends beyond using SER tags as prompts by leveraging the multi-head attention mechanism within the Transformer architecture. Contrary to other methods that utilize cross-attention, we did not employ an additional cross-attention module. Instead, we utilized the self-attention blocks within the multi-modal pre-trained model to perform the functions of cross-attention. This approach saves memory and computational time. Additionally, we leveraged the multi-head attention mechanism inherent in the self-attention block, which allows different attention heads to focus on varied content, offering mutual backup and reducing the incidence of missed recognition of entities.

  • Entity Classification and Entity Boundary Detection: Our work divided the task of SER on visually-rich documents into two sub-tasks: entity typing and span detection, and addressed both using the same framework. It has successfully addressed the task of entity extraction under the BIO tagging scheme that fits visually-rich documents. We have innovatively used natural language descriptions as prompts (beginning and inner) for span detection. In the context of NER on plain text, the demarcation of boundaries between entities has not been sufficiently emphasized. However, in SER tasks on visually-rich documents, distinguishing the boundaries of entities accurately with the BIO tagging scheme is crucial due to the possibility of two entities of the same entity type being positioned adjacently. If their boundaries are not correctly distinguished, they will be erroneously merged into a single entity.

  In conclusion, our work presents a novel approach to few-shot SER tasks on visually-rich documents, addressing unique challenges that have not been explored in existing plain text-based NER methods. We believe that these contributions significantly extend the existing body of knowledge on entity recognition tasks.

Question

• Question #1: Modifications on few-shot NER methods
  • Thank you for your suggestion. In Section 2, only a few methods we mentioned share the same In-Label-Space setting as ours, including COPNER and EntLM. These methods hold potential for applications on visually-rich documents. We have experimented with these methods, and the results have been included in Table 2 of our paper. The performance of these methods was found to be less satisfactory compared to ours, affirming the effectiveness of our proposed method. Given the constraints on the length of the main text, we initially did not provide detailed information about the modifications made to these comparison methods. However, to address your concerns, we have now updated our document and included the details of the modifications in Appendix A.2 of our paper.