UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition

1. Question on Entity Type Transformation:

In Section 3.1 of our paper, our goal is to construct a diverse dataset comprising various entity types. Given that these types are identified by ChatGPT and are not known in advance, it is crucial for ChatGPT to output both the entities and their corresponding types. Conversely, in Section 3.2, the focus shifts to instruction tuning using the dataset created in Section 3.1. At this stage, as the entity types are already known and explicitly included in the query (as in "What describes $t$ in the text??", where $t$ represents the entity type), there is no need for the model to output the entity type again. The model's task here is to respond to the query with appropriate entities related to the specified type, not to identify or output the type itself. This distinction in the model's output requirements between Sections 3.1 and 3.2 is an intentional design choice that aligns with the specific goals of each stage of our methodology.

2. Question on ELSE:

Thanks for the question. The issue with ELSE and other excluded types is that they serve as catch-all types in some datasets and have no clear definition. ChatGPT won’t be able to create appropriate examples for them due to the lack of definition of these ambiguous types. Note that even if we add the dataset name, ChatGPT still won’t be able to generate appropriate examples, as it’s not privy to the idiosyncrasies of the given dataset.

The dataset-specific instruction-tuning template just adds the dataset name in the prompt, which proves beneficial when supervised examples are available. This enables the model to discern fine-grained differences for the same entity type across various datasets. In the supervised setting (e.g., Table 2), we did include ELSE types. However, this won’t help the zero-shot setting, when we are using ChatGPT to generate labels.

3. Question on Figure 6

The main purpose of Figure 6 is to demonstrate the impact of employing dataset-specific instruction tuning templates on the performance, particularly in relation to label overlap. In this figure, we compare two variants of our UniNER model, both derived from LLaMA and trained on the exact same datasets. The only difference between these two models lies in using dataset-specific templates or not. Our analysis, as illustrated in Figure 6, suggests that the substantial improvement in performance for well-known datasets can be attributed to the effective reduction of label overlap through the application of dataset-specific templates. We believe that the observed improvements are not indicative of data contamination.

1. Comparison with ChatGPT's Performance

In knowledge distillation, it is not uncommon that the student model may outperform the teacher model, given that the teacher model is more general, whereas the student model just needs to focus on the specific tasks it is fine-tuned for (open NER in this case). While individual self-supervised examples may be noisy, with a large number of them, the student model may learn about the underlying cleaner patterns.

For instance, consider a recent paper on knowledge distillation for drug adverse event extraction, where the student model surpasses the GPT3.5 teacher by over 6 absolute points and nearly matches the supervised performance:

https://arxiv.org/abs/2307.06439
Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events.

Re using the UniversalNER prompting template with ChatGPT: We experimented with various prompting techniques for ChatGPT, selecting the most effective one to attain the strongest results. We did try the UniversalNER prompting template on ChatGPT, but it didn't perform as well, resulting in a drop of more than 4 absolute points in F1 compared to the best prompt. We ultimately used the prompt template from Ye et al., 2023, which yielded the strongest ChatGPT baseline among all prompt templates we compared.

2. Question about Entity Type Paraphrasing

We concur with your insight that this issue is particularly noticeable in true open-domain NER for entity types not known ahead of time. However, even for known types like "city", there may be sensitive issues, as demonstrated in Figure 8. Our hypothesis is that asking ChatGPT to generate definitions could improve recall by making it more robust to linguistic variations, while also introducing some noise that might negatively impact precision. We appreciate your suggestion and will include a discussion in our next revision.

3. Recognition of diverse entity types

Thank you for your suggestion. We conducted a manual evaluation on 100 samples, which indeed confirms the UniversalNER model's ability to recognize diverse entity types across different domains.

We uploaded these 100 samples to an anonymous folder and also provided five samples in a separate response (due to the space limit) for a quick demonstration.

We conducted a manual evaluation on 100 samples and confirmed that the UniversalNER model's ability to recognize diverse entity types across different domains. These samples were uploaded to an anonymous folder. Below we provided five samples for a quick demonstration:

Example 001

The aircraft took off from Saddam International Airport ( later renamed Baghdad International Airport ) in Baghdad , Iraq around 11:30 p.m. , flying to Gimpo International Airport in Gangseo-gu , Seoul , South Korea , with stops at Abu Dhabi International Airport in Abu Dhabi , United Arab Emirates , and Don Mueang International Airport in Bangkok , Thailand .

User: What describes airport in the text?
Assistant: ['Saddam International Airport', 'Baghdad International Airport', 'Gimpo International Airport', 'Abu Dhabi International Airport', 'Don Mueang International Airport']

Example 002

8 8 http://cosmologist.info/cosmomc/ The BAO data sets used in this work include 1. the isotropic BAO measurements using the 6dFRS (6dF; Beutler et al. 2011 ) and SDSS main galaxy sample (MGS) (Ross et al. 2015 ) at effective redshifts and , respectively; 2. the BOSS DR12 anisotropic BAO measurements at three effective redshifts (BOSS 3 z bin) in Alam et al. ( 2017 ) or at nine effective redshifts (BOSS 9 z bin) in Wang et al. ( 2017 ) and Zhao et al. ( 2017b ); 3. the eBOSS DR14 isotropic BAO measurement at (Ata et al. 2017 ); and 4. a combination of 6dF + MGS + BOSS 3 z bin + eBOSS DR14 (All 3 z bin), or a combination of 6dF + MGS + BOSS 9 z bin + eBOSS DR14 (All 9 z bin).

User: What describes Dataset in the text?
Assistant: ['6dFRS', 'SDSS main galaxy sample', 'BOSS DR12', 'eBOSS DR14']

Example 003

For this purpose , employee benefit plans generally include profit sharing , 401 ( k ) , and 403 ( b ) plans , but do not include : IRAs ; SIMPLE , SEP , or SARSEP plans ; plans covering self-employed individuals and their employees ( formerly Keogh/H.R. 10 plans ) ; health savings accounts ; or plans investing through the Fidelity Advisor 403 ( b ) program ; Insurance company separate accounts ; Broker-dealer , registered investment adviser , insurance company , trust institution and bank trust department managed account programs that charge an asset-based fee ; Current or former Trustees or officers of a Fidelity fund or current or retired officers , directors , or regular employees of FMR LLC or FIL Limited or their direct or indirect subsidiaries ( Fidelity Trustee or employee ) , spouses of Fidelity Trustees or employees , Fidelity Trustees or employees acting as a custodian for a minor child , or persons acting as trustee of a trust for the sole benefit of the minor child of a Fidelity Trustee or employee ; Purchased for any state , county , or city , or any governmental instrumentality , department , authority or agency ; Purchased by a charitable organization ( as defined for purposes of Section 501 ( c ) ( 3 ) of the Internal Revenue Code ) or a charitable remainder trust or life income pool established for the benefit of a charitable organization ; Qualified tuition programs for which FMR or an affiliate serves as investment manager , or mutual funds managed by Fidelity or other parties ; public and private retirement programs and non-U.S. insurance companies , if approved by Fidelity ; Destiny Planholders who exchange , or have exchanged , from Class O to Institutional Class of Fidelity Advisor funds .

User: What describes business in the text?
Assistant: ['FMR LLC', 'FIL Limited', 'Fidelity']

Example 004

The top selling hybrids in the U.S . market are the Fusion Hybrid with 127,572 units , followed by Escape Hybrid with 117,997 units , and the C-Max Hybrid with 54,236. , Ford is the world 's second largest manufacturer of hybrids after Toyota Motor Corporation , with 400,000 hybrid electric vehicles produced since their introduction in 2004 .

User: What describes car in the text?
Assistant: ['Fusion Hybrid', 'Escape Hybrid', 'C-Max Hybrid']

Example 005

The city 's parks department also operates two Par 4 18-hole public golf courses , 1 Par 3 18-hole course , one Par 4 9-hole golf course , two disc golf courses , Garvin Park , Lloyd Pool , the Goebel Soccer Complex , Swonder Ice Arena and the C.K .

User: What describes sports facility in the text?
Assistant: ['Lloyd Pool', 'Goebel Soccer Complex', 'Swonder Ice Arena']

We will include these examples in our final version.

1. Re Supervised Baselines

Thank you for suggesting an additional supervised baseline. We will include it and add discussion in the final version. However, we do want to point out that the suggested baseline from (Liu et al. 2022) was only evaluated on CoNLL and ACE-05, which are classic closed NER datasets. In contrast, our evaluation in Table 2 is based on a much broader collection of datasets introduced by the latest InstructUIE paper (Wang et al. 2023a), which we compared against and demonstrated a substantial performance gain.

In sum, the supervised results in the suggested Liu et al. 2022 paper were attained on much easier datasets with substantially narrower scope. They are not comparable to Table 2, which reported results on much more challenging datasets. UniNER attains the state-of-the-art accuracy and represents a large step forward compared to prior state of the art InstructUIE.

In general, it’s also important to emphasize that the true power for UniNER lies in open NER in the zero-shot setting, where no manual annotation or supervised labels are available. E.g., see Figure 5. This open NER setting is an important research frontier, as many real-world scenarios require recognition capabilities for a large number of entity types, for which supervised learning simply can not scale. E.g., in biomedicine, there are tens of thousands of diagnostic types alone, not to mention other medical concepts.

There may exist supervised baselines for a tiny fraction of the entity types, where there are prior efforts in annotating labeled data for supervised learning. However, for the vast majority of entity types in real-world applications, it’s simply infeasible to train supervised NER systems covering all of them, which would incur prohibitive cost in annotating sufficient number of labeled examples for each entity type. We agree that UniversalNER is far from perfect for many entity types, but as a zero-shot open NER system, it attains the state-of-the-art performance, and represents a big leap forward compared to existing generalist NER systems.

Similarly, it’s important to note that any training cost in data generation and distillation is dwarfed by the efforts to train supervised NER systems for all possible entity types.

2. Re Instability

To address potential instability in our data construction process, we set the temperature parameter to 0 during the data generation phase with GPT-3.5-turbo. Our initial exploration confirmed that this adjustment (i.e., temp=0) ensured consistent and reproducible outputs for our dataset. This approach mitigates the variability often associated with generative models and enhances the stability of our dataset, providing a solid foundation for our experiments.

We understand the importance of reproducibility in scientific research and are committed to facilitating it for our work. To this end, we plan to release the pretraining data and the necessary scripts for reproducing pretraining and inference, enabling other researchers to independently verify the effectiveness of UniversalNER. Additionally, our repeated re-training of the model has consistently reproduced the reported results. We believe these measures effectively address the reviewer's concerns regarding stability and reproducibility, and we hope this clarification demonstrates our commitment to maintaining the highest standards of scientific rigor in our work.

3. Re Collection of Passage-Query Pairs

Regarding the question about the collection of passage-query pairs, we followed a standard practice to ensure a balanced representation of datasets. Specifically, for each dataset containing more than 200,000 passage-query pairs, we subsampled 200,000 pairs to include in the instruction-tuning set. This approach prevents larger datasets from dominating the learning process. We will release our subsampled datasets to facilitate reproduction and ensure transparency.

Thank you for the valuable and insightful feedback. We appreciate your recognition of the importance and potential impact of the dataset we constructed. Please find our response regarding your concerns on reproducibility and dataset processing below.

1. Entity Type Selection and Removal

During the data preprocessing phase, we took several steps to ensure the quality and consistency of our dataset:

• We excluded any unparsable outputs identified by JSON parsing errors.
• We removed non-English entities and their associated types to maintain a consistent linguistic framework.
• We omitted entities classified under 'NA' types, such as none, NA, and else, since they are either ambiguous or undefined, hindering effective analysis.

2. Handling of 'ELSE' and Syntactic Types

We systematically remove all 'ELSE' types (e.g., else, NA, O, none, misc) from our dataset. This decision is based on the unsuitability of these types for zero-shot evaluation, due to their broad and often non-specific nature. Additionally, the WLP dataset contains semantic role labels (e.g., modifier, action), which we exclude to maintain a focus on semantically rich and contextually relevant entities.

3. Reproducibility and Data Release

We appreciate that the reviewer recognized our dataset as a major contribution. We plan to release it, along with the pretrained models and scripts necessary for reproducing pretraining and inference, upon paper acceptance. provide details on dataset processing in the appendix and include a list of the removed entity types.