Dynamic Demonstrations Controller for In-Context Learning

Thanks for your appreciation of our clearly written, easy-to-follow, and comprehensive ablation study. We answer your concerns as follows:

Q1: It would be interesting to discuss what causes the observations in the pilot experiments. For instance, in Figure 2, what aspects of the datasets cause the different optimal k for different datasets?

A1: Thanks for your constructive feedback. We speculate that adding a demonstration to a prompt will have two effects: (1) Providing more information to the prompt, resulting in improvement in performance. (2) Causing the distribution of the prompt to become more different from the pre-training corpus of LLMs, leading to difficulty in understanding the prompt and reducing performance. When more demonstrations are added, the direction of the change in performance depends on which effect is more influential. For different datasets and LLMs, when adding more demonstrations, the strengths of Effect (1) and Effect (2) are different, leading to the variation observed in pilot experiments and also causing the difference in the optimal k.

Q2: It is still unclear which one is more effective among demonstration selection and dynamic demonstration number selection.

A2: Thanks for your precious suggestion. As we demonstrated in our paper, KATE (a demonstration selection method) obtains more competitive performance when adapted to these ten classification datasets than our demonstration number selection method. However, KATE uses a default number to select demonstrations and is not able to decide how many demonstrations to select is good. Our method is orthogonal to KATE while also being complementary. Thus, combining two methods can achieve better performance.

Q3: In terms of the limitations, when will the method fail, and when will the method have good performance?

A3: Thanks for your valuable comment. Regarding the $D^2Controller$ method, some LLMs exhibit a minor decline in performance on the MPQA, SST-2, and MR datasets compared to the default setting. One possible reason is that these datasets have relatively shorter average demonstration lengths (shown in Table 6), leading to encoded semantic representations that contain less information. Thus, the similarities measured by IICScore based on these representations are inaccurate. In this case, selecting an appropriate demonstration number for these datasets may be more challenging. In contrast, most of the datasets with relatively longer average demonstration lengths perform well with $D^2Controller$.

Thanks for your appreciation of our useful topic, neat idea, and well-designed method. About the mentioned questions, our answers are as follows:

Q1. How does $D^2Controller$ compare to a simple baseline where k is optimized as a hyperparameter using a validation set?

A1: Thanks for your valuable comment. Following your suggestion, we randomly sample some examples (i.e., 100, 200, 300) to construct validation sets as a baseline for choosing k. The results are as follows (note that the results we report are the average performance of ten datasets):

Based on these results, we can observe that using these examples does not lead to the optimal choice of k, and almost all of the results are inferior to $D^2Controller$. This further underscores the effectiveness of using IICScore to select a small number of representative examples.

Q2: How does $D^2Controller$ work with classification tasks where options have no consistent meanings? e.g., BigBenchHard dataset. Could the author provide more insights on this?

Jane quited her job on Mar 20, 2020. 176 days have passed since then. What is the date today in MM/DD/YYYY?
(A) 09/12/2020
(B) 11/12/2020
(C) 12/12/2020
(D) 09/12/1961
(E) 09/17/2020

A2: Thanks for your inspiring comment. The BigBenchHard dataset mentioned above is essentially not a classification task but rather a QA (Question-Answering) task. A possible way to work with it is to convert the QA form into classification form. Specifically, we pair the question sentence with each answer and label the pair with correct answer as "yes" and others as "no". Thus, the task is transformed to a 2-class classification task and we can apply our $D^2Controller$ to the dataset.

Q3: How does $D^2Controller$ work when k<|c|?

A3: We are sorry that we did not understand exactly the meaning of the above comment. We try answering it:

According to our understanding, |c| denotes the number of examples for a class c. When k < |c|, we can certainly sample k-shot examples from each class. When k > |c|, we can sample all examples from each class. In this paper, all ten of our datasets meet the condition where k < |c|.

Q4. $D^2Controller$ measures data similarity in representation space. Did the authors compare different text encoders and see whether / how they affect $D^2Controller$?

A4: Thanks for your valuable feedback. In this paper, we use LLM as a text encoder in $D^2Controller$ to measure data similarity. Following your suggestion, we also tried another two text encoders (i.e., BERT-large and RoBERTa-large). The results are as follows:

We can observe that $D^2Controller$(BERT-large) and $D^2Controller$(RoBERTa-large) perform worse than $D^2Controller$ on most of the LLMs (except for OPT 13B), which verifies the superiority of using GPT-architecture LLMs as the text encoder to measure data similarity in representation space.

Q7. Could the authors provide some insights on why sometimes LLMs fail to benefit from more IC examples? Do stronger LLMs (e.g., gpt-4) suffer less from this?

A7: Thanks for your constructive feedback. We speculate that adding an in-context example to a prompt will have two effects: (1) Providing more information to the prompt, resulting in improvement in performance. (2) Causing the distribution of the prompt to become more different from the pre-training corpus of LLMs, leading to difficulty in understanding the prompt and reducing performance. When more IC examples are added, the direction of the change in performance depends on which effect is more influential. For some datasets on some LLMs, the Effect (2) is stronger, leading to performance degradation when more IC examples are included.

Besides, similar to Section 3, we have conducted pilot experiments with the GPT-4 model on five text classification datasets. Due to budgetary constraints, for each dataset, we use five different seeds to test the model's performance in the 1-shot setting, the default setting (4-shot), and $k_{max}$-shot setting. Note that the maximum input length of the GPT-4 model we use is 8192 tokens, so the maximum shot number for SST-5, CR, MPQA, RTE, and CB is 32, 128, 256, 32, and 16. The results are as follows:

From the perspective of a general trend, when the input increases from a 1-shot setting to $k_{max}$- shot setting, the accuracy improves on the CR, MPQA, and RTE datasets while declines on the SST-5 and CB datasets. Moreover, in the above table, the RTE dataset achieves the best performance in the default setting, rather than $k_{max}$- shot setting. Thus, increasing the number of demonstrations in stronger LLM like GPT-4 does not necessarily improve performance.

Q8: In Section 5.4.4, the authors mention that they get better performance with fewer demonstrations. Maybe a more straightforward way to present this is to report (on average) how many tokens their method queries an LLM, and how does that compare to prior work (default k).

A8: Thanks for your valuable suggestion. We reported the average number of tokens used by three methods (default k, maximum k, and ours) to query LLM:

Based on these results, we can observe that our method uses fewer tokens to achieve better performance compared to maximum k. Especially on some LLMs, such as Cerebras-GPT 2.7B and OPT-13B, $D^2Controller$ saves almost 30% and 50% tokens. Meanwhile, although our method uses more tokens compared to the default k, it achieves an average relative improvement of 5.4% on ten datasets.

Q9: About typos.

A9: Thanks for your careful review. We have fixed these typos in the revision.

I have read the authors response including those answering other reviewers' questions. I appreciate the authors' effort on clarifying things so I'm happy to raise my score a bit. However, please note, I give 6 -> 8 only because there is no option of 7. I don't think the current version is as mature as 8. (E.g., the authors have included quite a bit of new experiments during the rebutal period, mainly in the appendices. It may require some efforts to merge some into the main content, with some non-trivial rewriting.)

We are very happy to hear that the reviewer has raised the score! We thank you again for the invaluable feedback and insights. We really appreciate it!

Thanks for your appreciation of our excellent job, high-standard writing, and meaningful method. We address your concerns as follows:

Q1: About novelty and contirbutions.

A1: Thanks for your valuable comment. The core contribution of this paper comes from two aspects:

• Contribution 1: We refute the prevailing belief that as the number of demonstrations increases, ICL performance continues to improve. Instead, the optimal numbers of demonstrations vary on different datasets and LLMs;

• Contribution 2: Based on the above observation, we propose a method named $D^2Controller$ to dynamically select the number of demonstrations, which is highly scalable and can not only be applied to LLMs but is also compatible with previous ICL methods.

  The core contribution of $D^2Controller$ is a fresh perspective of selecting small-scale but comprehensive evaluation examples to provide an accurate evaluation of the demonstrations.

  • Novelty: To the best of our knowledge, previous work [1] [2] mentioned above both randomly sample examples as a validation set. Different from them, we construct small but comprehensive evaluation examples.

  • Implementation: To realize the idea, according to the characteristics of the classification task, we adopt the approach of "minimizing intra-class distance and maximizing inter-class distance". Of course, the approach is replaceable, as long as it can achieve our idea.

  • Effectiveness: To verify the effectiveness of evaluation examples selected by our method, we compare our method with using randomly sampled validation examples (With 3 different sizes, 100, 200 and 300). The results are as follows:

Based on these results, we can observe that $D^2Controller$ uses fewer examples but achieves better results compared to random sample examples.

Q2: Could the authors provide more information on the cost of “Evaluation Examples Selection” and “Accuracy-based Evaluation” stages?

A2: We appreciate your interest in understanding the cost of these stages. In the following, we provide running times for three different sizes of LLMs during the Evaluation Examples Selection and Accuracy-based Evaluation stages, respectively.

References:

[1] Chang, T. Y., & Jia, R. (2023, July). Data curation alone can stabilize in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8123-8144).

[2] Li, X., & Qiu, X. (2023). Finding supporting examples for in-context learning. arXiv preprint arXiv:2302.13539.

Thanks for your feedback. Regarding the $D^2Controller$, the core contribution is we propose a fresh perspective of selecting small but comprehensive evaluation examples to provide an accurate evaluation of the demonstrations. To achieve comprehensiveness, we co-opt the idea of the fit ability and the generalization ability in ML and adapt it to the ICL for the first time, to the best of our knowledge. As an initial attempt to implement the idea, we propose IICScore to guide us to select examples. The reason we say IICScore is "replacable" is because there may be other approaches can also achieve this goal, which leaves room for others to explore. Thanks for your response again.

Thanks for your appreciation of our sound methodology, make-sense questions, and appropriate baselines. We answer your concerns as follows:

Q1: What if one uses as many examples as possible for selecting k?

A1: Thanks for your valuable comment. Following your suggestion, we randomly sample more examples as a baseline to select k. Specifically, we construct three different sizes of validation sets (100, 200, and 300) to select k. The results are as follows (note that the results we report are the average performance of ten datasets):

Based on these results, we can observe that using more examples does not lead to the optimal choice of k, and almost all of the results are inferior to $D^2Controller$. This further underscores the effectiveness of using IICScore to select a small number of representative examples.

Q2: Why use fewer validation examples to select k? Is it because the number of available validation examples is limited?

A2: We are sorry that there may be a misunderstanding, because our method is not proposed for the situation where the number of available examples is limited.

We use fewer validation examples to select k mainly due to the following three reasons:

• Effective: As described in A1, using a small number of examples selected by $D^2Controller$ is more effective than using a validation set composed of a larger number of randomly sampled examples.
• Efficient: Our method is much more efficient than using a validation set. In the following Table, we report the running time of selecting k using $D^2Controller$ and using a validation set of 300 examples, respectively. Although the cost of hyperparameter selection would not affect the inference speed at test time, it is still worthwhile accelerating the selection process.
• Cheap: For LLMs that require monetary paid-for API (e.g., GPT-3), it is very expensive to use a large-scale validation set to select hyperparameters, especially for academia. Thus, a method that is effective while requiring less money is necessary.

As for the situation you mentioned in the review, although we do not design our method out of consideration that the available validation examples are limited, we believe our method would be competitive under the situation.

Q3: "To measure similarities, we transform each sentence x to a vector representation x, which essentially is a language modeling distribution, by querying LLMs with x and obtaining the output" - what does this mean exactly?

A3: Thanks for your valuable feedback. The intended meaning of this statement is that we input each sentence x into LLMs, thereby obtaining sentence vector representations. We have revised it to make it clear.