PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition

We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s novelty and writing. We are grateful for the opportunity to address the concerns raised.

W1-speedup is weak when only one entity type: Even if the entity type is one, if there are many mentions in that entity type, PaDeLLM can still improve inference speed through Step 2, because we predict all label-mention pair in parallel. As compared in the table, the string length of PaDeLLM still shorter than AR method (<mention x> is the templated added not generated by the model).

W2-reporting other method latency:

• Our primary comparison for inference efficiency is between two typical AR methods, AutoReg_aug and AutoReg_struct. Other methods, such as BINDER, are encoder-only models and are not comparable. Methods like GoLLIE and DeepStruct use larger LLMs and different prompt engineering, making direct inference speed comparisons unfair.
• We did not report the results for AutoReg_aug and AutoReg_struct in Table 3 because these methods are not immediately suitable for the zero-shot setting. Our preliminary experiments show that they have significantly higher latency compared to PaDeLLM, and their F-score is quite limited. We will include these experimental results in the updated version.

W3-report base model: We have added a column specifying the base language model used by each method and will update these tables in the next version. Except for GoLLIE and DeepStruct, all other methods are encoder-only, which results in a smaller number of parameters. GoLLIE and DeepStruct use backbones of 34B and 10B parameters, respectively, which are larger than PaDeLLM based on LLaMA-2 with 7B parameters.

W4-writing improvement

• Thank you for pointing out the typos. We will correct them in the updated version.
• The meanings of underline and bold formatting will be clarified in the updated version.
• The reason for the asterisk (*) is explained in the caption of Table 9. We will provide a more explicit explanation in the updated version.

W5-considering ICL: This is a good suggestion. We have acknowledged it in lines 287-289, and we will rephrase the wording to make it clearer. We'll also list caching mechanism as a potential area for future exploration (line 310).

Thanks for your responses.

W1-speedup is weak when only one entity type...

This may need to be acknowledged in your paper. Since only one mention also existing in many scenarios.

W3-report base model...

Thanks your adding this column. This is very important for readers to understand the cost of reaching the coressponding performance.

Though the decoding techniques could improve the efficiency. I still believe that using caching techniques and long-context ICL, i.e., fixed 1000-shots, may reach better performance but good efficiency. But I acknowledge that these techniques could be used in the decoding methods propoed in this paper.

Since most of my concerns have been addressed. I increase my soundness score to 3 and over accessment score to 6.

We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s contributions. We are grateful for the opportunity to address the concerns raised.

W1-loss aligment of mention to the tokens: We acknowledge that PaDeLLM loses token position information, and we recognize that incorporating token location (like start and end index) might enhance performance. We will consider this in future work, as suggested in [1].

[1] A Unified Generative Framework for Various NER Subtasks (Yan et al., ACL-IJCNLP 2021)

W2-dedpulication is aggressive: Yes, we acknowledge the de-duplicate strategy is aggressive. In our dataset, instances where two mentions appear under multiple labels are rare, as evidenced by our statistics. This is why we propose using the de-duplication mechanism. However, in real-world applications where a mention might be allowed to appear under multiple labels, we can choose not to use the de-duplication mechanism. This decision is a trade-off.

W3-improper wording: We will tone down the wording in the updated version.

Q1-large number of entity types or mentions:

• Theoretically, this method is effective regardless of the number of mention categories because each label-mention pair is handled by a different string. If you have more than 20 different mention categories, you can process them in parallel using over 20 different sequences to predict the number of mentions. Once that is complete, proceed to Step 2 to predict the label-mention pairs.
• In our in-house NER tasks, where there are often numerous mentions of a given type, we found that the method can manage tens of mentions without significantly losing accuracy. However, in practice, we typically limit the input to 512 tokens to prevent potential issues.

Q2-generating inconsistency: During inference, <mention x> is added to the prompt, which is not generated by the model. For instance, if the model outputs that there is 1 person at Step 1, we include <mention 1> in the prompt. If the model indicates there are 2 person at Step1, we duplicate the input string and add <mention 1> and <mention 2> to two strings, respectively. This approach effectively avoids inconsistencies. See Figure 2 caption the dashed box vs the solid box, and line 139-144.

We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s contributions, writing and novelty. We are grateful for the opportunity to address the concerns raised.

W1-token location information:

• NER traditionally relies on sequence labeling, where each token is assigned a class based on its position. This requires token location information. However, in the seq2seq approach such as PaDeLLM, Autoregressive_struct and autoregressvie_aug, token location information is not mandatory.
• For the sentence "Donald Trump owns the Trump Organization," PaDeLLM can differentiate and label the entities as "PER: Trump" and "ORG: Trump" if the de-duplication mechanism is not employed, but the token location information is losing. We recognize that including token location might improve performance and will consider this in future work, as suggested in [1].

[1] A Unified Generative Framework for Various NER Subtasks (Yan et al., ACL-IJCNLP 2021)

W2-drawback of de-duplication strategy: Yes, the de-duplication mechanism will filter this case. In our dataset, instances where two mentions appear under multiple labels are rare, as evidenced by our statistics. This is why we propose using the de-duplication mechanism. However, in real-world applications where a mention might be allowed to appear under multiple labels, we can choose not to use the de-duplication strategy. This decision is a trade-off.

We thank the reviewer for the thoughtful feedback and appreciate the recognition of our paper’s contributions. We are grateful for the opportunity to address the concerns raised.

W1-Justification of the importance of two-step prediction:

We conducted an additional experiment using one-step prediction, where all mentions of the same label are predicted in a single sequence. As shown in the results below, the inference speed of this approach falls between the two-step prediction and purely AR model, which is as expected. However, the prediction quality is lower than that of the two-step prediction. In other words, two-step prediction outperforms one-step prediction in both inference speed and prediction quality. We will include these new results in the updated version as strong justification for two-step prediction.

W2-Reason that AR model is worse: We acknowledge that AR methods, which rely on dependency tokens, offer certain advantages. However, they also encounter challenges in specific scenarios:

• Nested NER: For example, in the ACE dataset, the term "human" is nested within the "Human rights group." All AR methods struggle to identify this nested structure. In contrast, PaDeLLM, which decomposes from dependencies, effectively addresses this issue.

• Parse Errors in AR Output: Additionally, AR models can encounter format errors as shown below.
  • Overlong Example (MSRA): ((组织:邓小平同志治丧委员会),(名称:江泽民),...,(名称
    • This is due to the prediction being cut off mid-output.
  • Error Format (Resume): ((公司:环三)+...+(名称:null)+(籍贯:null)+(学历:null)+(国籍:null))
    • Incorrect use of "+" causes parsing issues.
  • Invalid Null Entries: ((组织:亚协会长),...,(组织:庆祝东方新年),(组织:null),(null))
    • Entries with "(null)" lacking an entity type lead to parsing failures.
• Training/test configuration: for AR models and PaDeLLM, we use the same training and testing configuration as reported in Appendix D (i.e., line 581-586)
• We will release all resources including inference results for the community to replicate and analyze the results. And we will add all this analysis to the updated version.

W3-same mention under multiple labels:

• We acknowledge the method loses token location information, which will be explored in future work.
• We have identified instances where the same mention is assigned different entity types in the ground truth, as shown in the table. Although such cases are rare, addressing this issue is important for real-world applications. In these situations, we can disable the de-duplication, allowing 'Washington' to be predicted as both 'LOC' and 'PER' simultaneously.

W4-need fine-tune:

• First, we tested the zero-shot generalizability of PaDeLLM, as detailed in lines 249-260, demonstrating its efficacy without further SFT to handle out-of-domain entity types, as appreciated by Reviewer XzjF.
• Additionally, we can employ resource-friendly fine-tuning techniques such as LoRA, P-tuning, and others to achieve the SFT.
• For larger LLMs, it is feasible to use training-free methods like in-context learning to enable the LLM to make the two-step prediction, as highlighted in line 287-289. This approach will be explored in future work. We will add this discussion to the updated version.

Q1: Such situation is too rare, the statistics of prediction is shown below:

Q2: this is discussed in Section 6, the primary focus of our experiments is the comparison of our proposed method with baseline methods (Auto_Struct and Auto_Aug). Given that these methods employ the same LLM as the base model, data contamination is unlikely to significantly impact the results.

We kindly request your acknowledgement of our reply, and are welcome to further discussions for your questions and concerns. We would be fully appreciated if you would consider to improve the rating. We look forward to your response.

Thanks for the further discussion and we are grateful for the opportunity to address the concerns.

W1-one-step pred :

• In "one-step baselines". all mentions under the same label are predicted in one single sequence. If there is no label related to any mention in the example, the result is empty. Please refer to the exact prediction example below. The latency of the slowest sequence is reported as the overall latency of one example. | Entity | Text | NER Result | |--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------| | ORG | <entity>ORG<text>2004-12-20T15:37:00 Microscopic microcap Everlast, mainly a maker of boxing equipment, has soared over the last several days thanks to a licensing deal with Jacques Moret allowing Moret to buy out their women's apparel license for $30 million, on top of a$ 12.5 million payment now. | NER result: ["Microscopic microcap Everlast", "a maker of boxing equipment", "their"] | | PER | <entity>PER<text>2004-12-20T15:37:00 ... million payment now. | NER result: ["Jacques Moret", "Moret", "their", "their women"] | | GPE | <entity>GPE<text>2004-12-20T15:37:00 ... million payment now. | NER result: [] | | LOC | <entity>LOC<text>2004-12-20T15:37:00 ... million payment now. | NER result: [] |

• The order of mentions is kept the same as in the ground truth, aligning with the data provided by the respective dataset.

W2-nested NER elaborate:

• Definition of nested structure: In the table (ground truth), the phrase 'highly diverged [Drosophila [homeodomain |DNA] |DNA]' shows a nested structure where 'homeodomain' (DNA) is nested within 'Drosophila homeodomain' (another DNA).

• Elaboration AR methods struggle with such hierarchies due to their reliance on linear dependencies. The table's results (Auto_Aug, Auto_struct) highlight the need for models to not only understand the input and effectively model long-range dependencies to produce the correct format to be parsed. In contrast, PaDeLLM simplifies nested dependencies by breaking the tasks down during the two-step prediction, allowing it to maintain a shorter output format, making PaDeLLM better suited for handling nested structures compared to AR methods.

We will add this discussion to the Appendix in the updated version.

W3:We agree that this is a fundamental limitation of our proposed method. We will explicitly discuss this in the Limitation section. Additionally, we will include the experimental results without de-duplication in Table 4 (currently only presented in Table 6), to ensure a fair comparison for readers in the updated version.

W4:Yes, the "zero-shot generalizability of PaDeLLM" refers to the zero-shot datasets, PaDeLLM model itself needs to be trained. We believe that the strong instruction-following capabilities of LLMs (especially those much larger LLMs) make it feasible to implement PaDeLLM through in-context learning or other prompt engineering techniques without additional training.

If this addresses your concern, we would be fully appreciated if you would consider to improve the rating.

Thanks for the responses. I have some follow-up comments/questions.

W1 Which decoding strategies were used in the "one-step" baselines? How do you handle the order, i.e., "LOC Italy England" vs "LOC English Italy."

W2 Can you elaborate more on "All AR methods struggle to identify this nested structure? In contrast, PaDeLLM, which decomposes from dependencies?" Why is dropping dependency free from errors for the nested structure? (Additionally, can you provide a more formal definition for this nested structure?)

W3 Thanks for providing the statistics. However, I feel this is a fundamental limitation (in terms of capability) of the proposed method and should be addressed explicitly in the paper (at least in the Limitation section). Having post hoc refinements could be a solution, but they may also be applied to other methods, which would not be a fair comparison.

W4 Just to confirm: the "zero-shot generalizability of PaDeLLM" refers to the zero-shot datasets, right? The PaDeLLM model itself needs to be trained. Do you think you can make PaDeLLM a prompt-only LLM-based method?