IntentGPT: Few-Shot Intent Discovery with Large Language Models

We appreciate the reviewer's time to read our paper and provide feedback. We are glad that the reviewer found the paper to provide good guidelines for how to prompt GPT-4 and discover intents without any training overhead. We respond to the comments in detail below.

Tested on only 2 datasets. Please perform experiments on more datasets that have diverse input structure.

We picked those two datasets for our paper because they're widely recognized as the standard benchmarks in open-set intent discovery [3, 4, 5], which makes it fair to compare our work to existing methods. However, the comment inspired us to extend our results to three new datasets. First, as was highlighted by reviewer kRkh, we have run our method on a multilingual dataset (MTOP) [6] and presented the results in our response to them. Please see our comment there for results on the MTOP dataset. To further strengthen our results we consider two additional datasets, namely SNIPS [1], and StackOverflow [2]. These datasets can be interesting because they show a much reduced number of categories, hence will allow us to judge the robustness of our model in a regime with a small number of categories. The results are shown as follows:

We compare IntentGPT-3.5 and IntentGPT-4 against available public results on two of the baselines presented in the paper, namely DSSCC [7] and CDAC+ [8]. Our model outperforms these methods except DSSCC on StackOverflow, which performs better for the ACC metric. These results required no human intervention in modifying the prompts, or training, showcasing the robustness of IntentGPT.

Prompt and few shot example usage might change depending on the data.

While the IntentGPT prompt and few-shot examples might change across datasets our model is able to automatically adapt the prompts and in-context examples to a given dataset, hence generating more informative and valuable prompts. Furthermore, it does not require human effort when porting the model into a new dataset, as it is fully automated for generating the needed prompts given the new dataset, capturing the relevant nuances about the data domain.

Known intent feedback might not be usable for zero shot context.

Our Known Intent Feedback technique is indeed usable in a zero-shot setting. In this setting, we inject the discovered intents back into the prompt, starting from an empty database. In fact, we have experiments available on the paper for the zero-shot setting case. We show the results of our method in Table 1 on the paper (see results for the unsupervised setting), and also in the first row of ablations in Table 2, for both GPT3.5 and GPT-4 on the two datasets.

We thank the reviewer for reviewing our paper. We are encouraged to read that the reviewer finds our paper is attacking a difficult problem, and our methodology performs well compared to previous baselines. We now respond to their questions below.

The Semantic Few-shot Sampling (SFS) component reads to me as Retrieval Augmented Generation or RAG, then is not a novel contribution.

The main difference is that Retrieval Augmented Generation (RAG) has been mainly explored in knowledge-intensive tasks like Question Answering [1], where factual information is retrieved to ground the generation, while we use it to extract few-shot examples, a much less explored problem [2, 3]. Therefore, we would rather consider the SFS and SKIF components as minor contributions to the paradigm of few-shot learning building upon the RAG model. We have updated our paper with this discussion, citing relevant work on RAG.

[1] Lewis, Patrick, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33 (2020): 9459-9474.

[2] Liu, Jiachang, et al. "What Makes Good In-Context Examples for GPT-$3$?." arXiv preprint arXiv:2101.06804 (2021).

[3] Izacard, Gautier, et al. "Few-shot learning with retrieval augmented language models." arXiv preprint arXiv:2208.03299 (2022).

Why not test other prompting schemes like Chain of Thought?

Chain of Thought is an appealing research avenue in extending our research. Previous works that use Chain of Thought [1, 2], improve in-context learning capabilities by performing two forward passes: one for generating a rationale of the task and a second to perform the task using that rationale. We perform a similar approach by first using an LLM to generate an informed and domain-specific prompt, and then use it to perform the task with a second LLM. We coincide with the reviewer that exploiting this prompting scheme could further improve results. We have incorporated this direction for future work in the revised paper.

[1] Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022).

[2] Lee, Harrison, et al. "Rlaif: Scaling reinforcement learning from human feedback with ai feedback." arXiv preprint arXiv:2309.00267 (2023).

I am not convinced that the ICPG step is producing better prompts than what a human would give.

As seen in the generated prompts from the ICPG module (see Appendix A.2, Prompts 2 and 3), it creates new prompts with rich information about the dataset at hand, which is different from the “vanilla prompt” that a human can create (like the one seen in Appendix A.2. Prompt 1). As shown in responses to reviewers GqJc and NWU5 about the ablation studies depicted in Table 2, the ICPG module is beneficial for obtaining performance boosts. Finally, let us show how the ICPG module creates better prompts than humans by testing it in a multi-lingual dataset, where a human might not know all the languages to manually create prompts. We have extended our experiments to the MTOP [1] dataset, and the results depict how IntentGPT's ICPG module generates a new prompt in the desired language automatically (see response to reviewer kRkh).

[1] Li, Haoran, et al. "MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark." arXiv preprint arXiv:2008.09335 (2020).

What is the “simple prompt description” that was done in the ablation study of the ICPG module?

The simple prompt description is included in Appendix A.2 (Prompt 5: E. Basic human-generated task description) to act as a baseline for our ablation analysis.

Did you consider the inclusion of test examples that express no intent or multiple intents?

In our experiments, we did not test instances that may not have an intent or have multiple intents associated. Nevertheless, to handle multiple intents, our method could be easily extended to handle multiple or no intents, by adjusting the original prompt description feed to the In-Context Prompt Generator and formatting the examples to support multiple intents in a single response. However, there are no public benchmarks for multiple intent or no intent, but we consider this as a very interesting task for future work.

Thank you for your thoughtful feedback. We are happy to see that the reviewer sees benefits to our Known Intent Feedback (KIF) mechanism and that we provide a comprehensive analysis of model hyper-parameters. We respond to the reviewer’s comments in detail below.

The idea of using LLMs for few-shot learning is not highly creative or novel

While LLM-based few-shot learning has received significant attention in recent literature[1, 2], the challenge of generating effective prompts for LLMs remains unsolved [3]. We recognized an opportunity to improve existing methods by proposing two valuable contributions. First, we introduce the use of an In-Context Prompt Generator (ICPG) LLM that can generate dataset-specific and informative prompts (ICP). Second, we introduce the technique of Known Intent Feedback (KIF) that can be extended for the general tasks of open-world class discovery. For the task of Intent Discovery, we see that these techniques can boost performance for our method, outperforming previous baselines using substantially less data, no training overhead, and minimal human effort.

Is Known Intent Feedback (KIF) unfair to previous deep learning approaches?

We respectfully disagree. KIF is not unfair compared to previous methods as it does not provide our approach with unfair information. KIF works as a regularization in that it restricts the amount of discovered classes by making sure they are separate enough from each other.

The research is limited to English datasets. Can IntentGPT work with multilingual datasets?

We chose the English datasets since they seem to be the standard benchmarks in recent papers for the open-set intent discovery task [4, 5, 6]. However, your comment inspired us to test this method in a different language. IntentGPT can be seamlessly used in multiple languages supported by the underlying language model. Therefore, we ran our method on the MTOP [7] dataset which consists of six languages: English, German, French, Spanish, Hindi, and Thai, and the results are shown below.

Table: Results on MTOP dataset

We observe that IntentGPT is able to generalize to languages other than English without requiring extra human effort. The first LLM in charge of generating the prompt (ICPG) seems to perform well at automatically generating the prompt in the desired language (see next comment to observe the prompts generated by ICP in Spanish, French, and Hindi).

Furthermore, quantitative results in the task of Intent Discovery displayed in the table show that our method works well in this multilingual setting, showing metrics in a similar range to the ones seen in other benchmarks, except for the case of Thai, where the GPT3.5 and GPT4 models struggle to handle. We leave as future work the comparison of IntentGPT with existing methods in this benchmark since existing methods do not report results on these datasets. We believe that these results reinforce the idea that IntentGPT is a robust approach for open-world class discovery tasks, and the novelty of the ICP and few-shot sampling techniques proposed. We have incorporated this new setting in the Appendix, including details on the automatically generated prompts.

[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 2020.

[2] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021.

[3] Lingyu Gao, Aditi Chaudhary, Krishna Srinivasan, Kazuma Hashimoto, Karthik Raman, and Michael Bendersky. Ambiguity-aware in-context learning with large language models, 2023.

[4] Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. Discovering new intents with deep aligned clustering. In AAAI Conference on Artificial Intelligence, 2021b.

[5] Rajat Kumar, Mayur Patidar, Vaibhav Varshney, Lovekesh Vig, and Gautam Shroff. Intent detection and discovery from user logs via deep semi-supervised contrastive clustering. In PConference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022.

[6] Xiang Shen, Yinge Sun, Yao Zhang, and Mani Najmabadi. Semi-supervised intent discovery with contrastive learning. In Workshop on Natural Language Processing for Conversational AI, 2021.

[7] Li, Haoran, et al. "MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark." arXiv preprint arXiv:2008.09335 (2020).

We thank the reviewer for taking the time to review our paper very carefully. We are pleased to find that the reviewer feels the presentation of the paper and literature review were quite good, and the substantial appendices included are helpful for the reader. We now address their concerns regarding the contribution of our work.

Does the first LLM, In-Context Prompt (ICP), contribute to any value to the model?

Yes, the first LLM, In-Context Prompt Generator (ICPG), adds value to our model. The setup described by the reviewer is present in the paper. Please see our ablation studies, detailed in Table 2, which shows a comparison between the use of original prompts with auxiliary data (a single LLM with few-shot (FS) and Known Intent Feedback (KIF)) against our ICP-enhanced prompts.

Specifically, Rows 2, 3, and 4 of Table 2 of the paper (for both GPT3.5 and GPT4 versions) show the baseline performance using a single LLM, while Rows 5 and 6 highlight the improvements with the ICP module activated. The results show a consistent performance gain of 1-2 points across both datasets and various GPT models, underscoring the efficacy of our approach.

Additionally, thanks to a new experiment run on a multi-language dataset suggested by reviewer kRkh, we can show how the first LLM is able to handle the language adaptation without human effort, only relying on a few context examples from the train set. We find this case relevant for highlighting the contribution of ICP.

The choice of SentenceBERT to compute embeddings is a strange choice. Did you try OpenAI’s GPT3+ models?

We tested text-embedding-ada-002 embeddings and we observed that they perform worse than SentenceBert (see table below). The most likely reason for OpenAI embeddings being worse is that it is primarily used to encode large contexts like documents for document retrieval, whereas SentenceBert is trained to efficiently encode short contexts, or sentences, like the size of our intents. We have incorporated this result in the Appendix of the paper.

How does performance change if only the feedback part is ablated (i.e. the database of intents is constant) from the entire pipeline?

We have run this experiment and present the results below.

Here, we study the impact of maintaining a constant intent database, only composed of the known intents at initialization, and not growing the database during the evaluation. We observe a degradation of the performance in all metrics, compared to rows 2-7 in the ablation table (consider these rows also in GPT-4). However, this setting outperforms the baseline shown in row 1 of the ablation, as we include the mechanism of injecting Intents into the prompt. We also display NDI, which explodes due to the lack of intent reuse. We have added the findings from this additional experiment to our ablation study in the paper.

The optimal clusters (155 and 88) are so close to the ground truth (150 and 70). Is this an indication of dataset leakage?

We do not use GPT-4 or a trained model to determine the optimal number of clusters. We utilize DBSCAN on the predicted intent embeddings to determine the optimal K for k-means clustering. Therefore, there can not be dataset leakage.

Sometimes the predicted intents match exactly the ground intent. Is that evidence that GPT4 knows a lot more than just the name of the datasets?

We thank the reviewer for taking the time to think about this problem in our Intent Discovery setup. We find that this concern is valid. Therefore, we propose the Frechet Bert Distance as a reasonable measure to compute the similarity between the predicted intent and the ground truth distributions to identify whether the model is regurgitating its predictions from memorization. We see that there is an acceptable level of dissimilarity between the real and predicted intents, with a score of 0.54. A perfect match would be 0 and a complete dissimilarity would be 1. As shown in the paper, we are not able to reproduce utterances or labels from the test sets by manually interacting with the model.

Notably, the problem of detecting pre-train data in LLMs has recently received attraction [1, 2], however, there is not yet a standard approach on how the decide that a text example was part of the pre-training data. We will explore this direction in future work.

[1] Shi, Weijia, et al. "Detecting Pretraining Data from Large Language Models." arXiv preprint arXiv:2310.16789 (2023). [2] Meeus, Matthieu, et al. "Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models." arXiv preprint arXiv:2310.15007 (2023).

It would be nice to have a longer version of Appendix A.7 with more examples and details about the statistics of discovered intents. How close are they semantically/syntactically to the ground truth labels?

We have included a complete list of results in the supplementary material (see results.csv). We also extended Appendix 7 with a study on the complete sets of discovered intents and ground truth intents. In that study, we compute SentenceBert embeddings from the two sets and visualize them using TSNE in a 2D grid. We observe that some intents coincide, and many others do not. The ones that coincide correspond to samples with a straightforward intent naming choice, while the others require more naming creativity. Note that the required capability for a good performance is to use intents robustly, assigning the same intent to utterances with the same ground truth intent, regardless of the exact name.

In the extended list of discovered intents, we also observe that there is a reasonable difference between the predicted intents and the ground truth despite their semantic similarity. We consider that IntentGPT-3.5 and IntentGPT-4, conditioned with the proposed prompt scheme, offer sufficient generalization capability to not solve the task by memorization.

We thank the reviewer for their time spent reviewing the paper. We are encouraged to read that you see our method has useful downstream application potential and outperforms existing baselines. We now respond to their comments in detail below.

The In-Context Prompt (ICP) seems to add little value. From row 3 vs row 5 in the bottom panel of Table 2, it seems that on average ICP led to no performance boost.

While the difference between rows 3 vs row 5 does not lead to a significant average performance boost, the scores between rows 4 and row 6 when adding ICP (both using SFS) indicate that it boosts performance. Notably, comparing rows 2, 3, and 4 (which do not use ICP) with rows 5 and 6 (which use the ICP), as a whole, shows a consistent improvement between 1-2 points in all the settings (across datasets and GPT models), except for the case of ARI on CLINC dataset for GPT3.5, where results remain similar (see also responses to reviewers NWU5 and kRkh on this matter). As this is an ablation study, a component may not boost performance in all scenarios but when combined with the other parts of our pipeline we show that ICP does help. ICP aims to redirect the prompt towards a specific domain without human labor, offering a boost in performance as seen by our ablations.

Is row 1 of Table 2 the right baseline to compare with? Iit seems that for row 1, the KIR ratio is 0 while the remaining rows have a KIR of 0.75.

As noted by the reviewer, row 1 in the ablation studies from Table 2 displays the zero-shot setting, which implies a KIR of 0, while others use a KIR of 0.75. While this comparison may be not well-suited for judging performance on the main metrics (i.e. NMI, ACC, ARI) it is interesting to observe how the Number of Discovered Intents (NDI) explodes. This is because this simple setting performs as a zero-shot, and there’s no regularization of the label space introduced by the Known Intent Feedback technique. Additionally, this setting shows how other metrics downgrade because of it. We have updated the paper by specifying that this row shows a zero-shot setting, and clarifying that a more fair comparison in terms of Known Intent Ratio can be done starting from rows 2-7.

How does 50-shot vanilla GPT4 with KIR = 0.75 perform? Is that closer to row 3 of table 2 or row 0?

A similar question was asked by reviewer NWU5, where they asked for 10-shot vanilla GPT4 with KIR = 0.75. This ablation is shown in rows 2, 3, and 4 of Table 2 in the paper, which presents the performance boosts of including the ICP module when comparing against rows 4 and 6 (Please see our comment to reviewer NWU5 highlighting the results). Further, to answer your question, we also ran an experiment with 50-shot vanilla GPT4 and KIR=0.75. We present the results as follows:

Comparing this result with Table 2, this setting would perform similarly to rows 2 and 3, and far from the top positions. Even though this setting uses a lot of few-shot samples, it is not able to get better results because of the vanilla prompt. Note that when adding the ICP to the 50-shot setting, we obtain our best results, with an NMI=96.06 on CLINC, showing the contribution of having the ICPG component.