Active Prompting with Chain-of-Thought for Large Language Models

Q4. Iterative Process and Uncertainty

Thanks for your questions! Conventional active learning is indeed an iterative process, but now some work feels that the iterative method is costly. So, there are also some recent studies that are one-pass [1,2]. To explore the effects of modification, we conduct more analysis by comparing the changes in terms of confident questions set before and after modification. Here, the confident question set is defined as a set of questions that the model is highly consistency (uncertainty is 0). With gpt-3.5-turbo, using the unmodified CoT, there are 325 questions in the confident question set (set X_1). After modification, with the modified CoT, there are also 325 questions (set X_2). Over 70 % questions in X_1 appear in set X_2. Although the uncertainty has increased in a small number of questions, we find that their uncertainty still remains very low.

Q5. Cost Compared with Auto-CoT

Auto-CoT applies Sentence-BERT to encode all the questions into vector representation and then adopts k-means clustering to produce k clusters of questions. Active-Prompt does not require CPU and GPU computation because it is purely based on ChatGPT API which costs money. Although there is a clear performance advantage compared with Auto-CoT (as shown in Appendix D), there is no clear advantage compared with Auto-CoT because of the different implementations. Thank you for your question!

Q6. Performance Drop in Figure 2

Your observation about the accuracy decrease in the SingleEq dataset is astute. We attribute this to potential gaps during the transfer process from GSM8K to SingleEq, likely caused by noise and fluctuation in the inference process. To address this, we conducted additional experiments and updated Figure 2 with more stable and consistent results. Due to the unavailability of the code-davinci-002 model, we switched to the text-davinci-003 model for these experiments. The revised figure now reflects a more consistent increase and convergence in accuracy, which we believe more accurately represents the performance of our method.

Once again, thank you for your valuable feedback, which has significantly contributed to the refinement of our work.

[1] Lin, Yong, et al. "Optimal Sample Selection Through Uncertainty Estimation and Its Application in Deep Learning." arXiv preprint arXiv:2309.02476 (2023).

[2] Wang, HaiYing, Rong Zhu, and Ping Ma. "Optimal subsampling for large sample logistic regression." Journal of the American Statistical Association 113.522 (2018): 829-844.

Dear Reviewer 1P1b,

We deeply appreciate your comprehensive review and insightful feedback. Below are our responses to your concerns:

Q1. Need for Additional Corpora

From the Table 1, we find that the generalization performance is correlated to the backbone models. If we look into text-davinci-002 and code-davinci-002, we will find the performance boost on GSM8K and AQuA is larger than asdiv, svamp, and singleeq. However, If we look into text-davinci-003 and gpt-3.5-turbo, we will find the performance improvement is similar. For example, with gpt-3.5-turbo, the improvement is around 1.5 for these two scenarios. Therefore, considering the generalization, we can still take the prompts selected by a similar domain / task and then apply them to unseen domains / tasks.

Q2. Differences in Uncertainty Measurement Methods

Thank you so much for your question! We strongly agree with your observation. In addition, we find entropy-based methods are generally better than disagreement-based methods. For example, across 8 downstream tasks and two models (text-davinci-002 and code-davinci-002), Active-Prompt (E) outperforms Active-Prompt (D) 11 times out of 16. Active-Prompt (D) is better at SVAMP and CSQA. Because the calculation of entropy takes the frequency into consideration, it provides a more accurate estimation for uncertainty. For the performance disparities between Active-Prompt (E) and Active-Prompt (D) across different datasets, it is because entropy could capture more fine-grained information to help estimate the uncertainty. Let’s take an example for illustration. We query the model 5 times and obtain the follow results:

For question 1, answers A1 = {2, 3, 3, 3, 6}

For question 2: answers A2 = {2, 2, 3, 3, 6}

The disagreement of A1 is 3 and the disagreement of A2 is also 3. However, the entropy is different. The entropy of A1 is 1.37 while the entropy of A2 is 1.52. This explains why entropy-based methods are better than disagreement-based methods across 11 out of 16 tasks. Because of the fine-grained information, entropy could distinguish the uncertainty of different questions better.

Q3. Costs

For the cost of selection, compared with selecting questions by humans, our proposed method is more efficient. For a new task, users need to do trials and errors a lot of times which costs a lot of human effort with unstable performance. Even so, the selected questions are still suboptimal. Second, as mentioned in Section 3.3, we limit the size of candidate instances to 1,000 which greatly reduce the cost. 1,000 is a good balance between cost and performance. We verified that with more than 1,000 instances, the performance would converge. Doing uncertainty estimation 10 times with a pool of 1,000 questions is acceptable. The cost is smaller than self-consistency, which usually requires 40 times inference, although it is an orthogonal technique and can be complementary to ours. In addition, inspired by the new experimental results requested by reviewer Wb4R, we are excited to find that questions selected by smaller models (e.g., Llama) perform well with larger models (e.g., gpt-3.5-turbo). Considering models like Llama are open-source which does not cause API cost, one may use it (with GPU) to replace black-box API.

For the cost of annotation, annotating 8 questions is not costly. Humans could finish it in several minutes. Furthermore, like you suggested, one can use ChatGPT or GPT-4 for annotation, which is much more easier. We tried some examples and found the GPT-4 annotations make a lot of sense and we believe they work well as human’s annotation. We are doing this and will add this experiment very soon (before Nov. 22). Thanks!

Dear Review 1P1b,

We sincerely appreciate your insightful feedback once again. In response, we have diligently addressed each of your concerns and queries. Key updates include:

1. Discussion about the Generalization Performance
2. Analysis of the Differences in Uncertainty Measurement Methods
3. Discussion about the Cost
4. Revisit to the Performance Drop in Figure 2

Would you be able to spare some time to review our revised response and confirm if it adequately addresses your questions? We are eager to engage in further discussions and provide additional clarifications on any new queries you may have. Thank you very much!

Dear Reviewer hZjq,

We greatly appreciate your thorough review and insightful feedback. Here are our responses to your concerns:

Q1. Generalization to New Tasks

Active-Prompt is designed to be versatile and can be effectively utilized even with a limited number of unlabeled questions. This flexibility allows for efficient generalization to similar tasks. For situations where collecting new questions is feasible, our uncertainty estimation method facilitates the selection of high-quality examples. In cases where question collection is challenging, transferring questions from a similar domain is a viable alternative. Our results in Table 1, showcasing the transferability of selected questions from GSM8K to other datasets like ASDiv, SVAMP, and SingleEq, illustrate this point effectively. This evidence supports the notion that Active-Prompt is not only effctive but also generalizable to new tasks.

Q2. Performance of Variance-based and Self-confidence-based Method

Regarding the variance-based method, its applicability is indeed limited to arithmetic tasks, as it struggles with textual responses. For the self-confidence-based metric, as discussed in Section 5.1 [Effects of Uncertainty Metrics], we observed its tendency towards overconfidence, leading to suboptimal performance. Consequently, our focus shifted to disagreement and entropy metrics, which demonstrated more reliable results. We have clarified these points in our manuscript to ensure a better understanding of our methodological choices. Thanks!

Q3. Accuracy Decreasing in Figure 2

Your observation about the accuracy decrease in the SingleEq dataset is astute. We attribute this to potential gaps during the transfer process from GSM8K to SingleEq, likely caused by noise and fluctuation in the inference process. To address this, we conducted additional experiments and updated Figure 2 with more stable and consistent results. Due to the unavailability of the code-davinci-002 model, we switched to the text-davinci-003 model for these experiments. The revised figure now reflects a more consistent increase and convergence in accuracy, which we believe more accurately represents the performance of our method.

Thank you once again for your constructive feedback, which has been invaluable in enhancing the quality and clarity of our work.

Dear Review hZjq,

We sincerely appreciate your insightful feedback once again. In response, we have diligently addressed each of your concerns and queries. Key updates include:

1. Analysis of the Generalization to New Tasks
2. Discussion about the Performance of Variance-based and Self-confidence-based Method
3. Revisit the Accuracy Decreasing in Figure 2

Would you be able to spare some time to review our revised response and confirm if it adequately addresses your questions? We are eager to engage in further discussions and provide additional clarifications on any new queries you may have. Thank you very much!

Dear Reviewer Wb4R,

We are grateful for your comprehensive and insightful review. Your feedback has been instrumental in refining our manuscript. Below, we address each of your points in detail:

Q1. Performance of Complex-CoT

We sincerely appreciate your suggestion to compare the performance of Complex-CoT with our proposed Active-Prompt methodology. As recommended, we conducted a detailed evaluation using the GPT-3.5-turbo-0613 model. The prompts for Complex-CoT were sourced directly from the original paper to maintain consistency in comparison.

We find that Active-Prompt consistently outperforms Complex-CoT across all tasks. In addition, we can combine the uncertainty and the complexity together to achieve a better performance, and we look forward to exploring the combination in future work. We have updated our draft with these new results in Appendix E. Thanks very much for your constructive suggestion!

Q2. Performance of Weaker Models

We acknowledge the importance of evaluating performance across models of varying capabilities. Therefore, we conduct new experiments with Llama2-13b-chat and Lllama2-70b-chat. The results are shown as follows.

Firstly, because smaller models do not have a good chain-of-thought ability, it is difficult for Llama2-13b-chat to understand the chain-of-thought prompting, with poor performance of both CoT and Active-Prompt. Especially, we find that the more complex the prompt, the poorer performance. However, with Llama2-70b model, our proposed Active-Prompt outperforms CoT by a large margin, demonstrating this method is still useful for weaker models.

Note that we are using the instruction-tuned version of Llama2-70b in all our experiments (i.e., Lllama2-70b-chat) because it is able to understand complex chain-of-thought prompting and follow human instructions.

Q3. Transferability between GPT and Llama models

Thanks for your suggestion! Your suggestion prompted us to investigate the transferability between GPT and Llama models. Because smaller Llama models perform poorly on reasoning tasks, we conduct experiments with Llama2-70b-chat. We conduct two types of experiments: (1) select questions by gpt-3.5-turbo and infer by Llama2-70b-chat and (2) select questions by Llama2-70b-chat and infer by gpt-3.5-turbo. Note that we are using the 0613 version of gpt-3.5-turbo. The results are shown as follows:

The model before the arrow denotes the model for actively selecting questions while the model after the arrow denotes the model for inference. The results demonstrate the feasibility of selecting questions with smaller models (Llama2-70b-chat) and applying the selected questions to larger models (gpt-3.5-turbo). In addition, selecting questions with larger models and applying them to smaller models has better performance.

Dear Reviewer Wb4R,

We are deeply grateful for your valuable insights during the initial review of our ICLR submission. Based on your feedback, we have meticulously revised our paper. Key updates include:

1. A comparative analysis with Complex-CoT.
2. Experiments incorporating weaker models.
3. An exploration of transferability between GPT and Llama models.
4. Enhanced clarity in our writing to prevent potential misunderstandings.

Would you be able to spare some time to review our revised response and confirm if it adequately addresses your questions? We are eager to engage in further discussions and provide additional clarifications on any new queries you may have.

Thank you very much for your time and consideration.

Dear Reviewer Wb4R,

Thank you very much for your insightful feedback!

• For the performance with “weaker” models, it is reasonable to see the results may not match those of “stronger” models. As highlighted in prior research [1], complex reasoning tasks such as GSM8K have been challenging for language models, often necessitating larger model sizes. Given that understanding the Chain of Thought (CoT) is difficult for "weaker" models, the observed results align with expectations. It is important to note that our original manuscript did not assert the applicability of our method to weaker models; our primary focus is on enhancing performance in larger models. The notable performance of our method with the Llama-70b model underscores its effectiveness in open-source models as well with an improvement of 1.9%.

• In terms of the prompts for SVAMP and SingleEq, they are derived from the GSM8K dataset and directly utilized in inference. The less favorable outcomes in these two datasets likely stem from the limited generalization capability of smaller model selections. Nonetheless, the substantial improvement on GSM8K (4.5%) demonstrates our method's strong in-domain transferability, although its out-of-domain transferability merits further exploration. We appreciate your suggestion and will incorporate these findings and modify our claims to make them more precise in our manuscript.

• Regarding the cost, we categorize it into two segments: the cost of question selection and the cost of CoT and answer annotation. Initially, in the absence of Active-Prompt and Complex-CoT, manual question selection is required, which constitutes the "human effort" we previously referenced. As for the cost of CoT and answer annotation, nearly all methods, including the baseline CoT [2], entail some cost, so our approach does not introduce additional expense in this regard. Moreover, we are trading a minimal cost in question selection for notable performance gains. Several enhancements in CoT also involve similar trade-offs, such as the incorporation of self-consistency [3] and clustering [4]. For your reference, the cost for selecting from over 1000 questions with gpt-3.5-turbo is around 5 US Dollars after inferencing 5 times for each question. We will report the details of the cost in the Appendix of our manuscript. Thanks!

• For the prompts of the three datasets (AsDIV, SVAMP, SingleEq), we utilize the same ones as for GSM8K, consistent with the approach in [2]. This practice of transferring GSM8K prompts to these datasets is a standard approach, widely adopted in subsequent CoT research, including ours. The Complex-CoT's prompts are available at https://github.com/FranxYao/chain-of-thought-hub/blob/main/gsm8k/lib_prompt/prompt_hardest.txt

We are truly grateful for your time and valuable contributions to this dialogue. Your constructive engagement has significantly enriched our understanding and focus on the model's additional features, which we believe do not detract from our method's primary experimental performance enhancements. We will incorporate the outcomes of these discussions into our revised manuscript and refine our statements regarding transferability for greater precision. Thank you!

[1] Wei, Jason, et al. "Emergent abilities of large language models." Transactions on Machine Learning Research (TMLR), 2022

[2] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in Neural Information Processing Systems 35 (2022): 24824-24837.

[3] Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." The Eleventh International Conference on Learning Representations. ICLR 2023

[4] Zhang, Zhuosheng, et al. "Automatic chain of thought prompting in large language models."The Eleventh International Conference on Learning Representations. ICLR 2023

Dear Reviewer DkmB,

Thank you very much for your comprehensive review and valuable feedback! We address your comments one by one as follows:

Q1. Performance in Table 1

Thank you for highlighting the discrepancies in the performance of gpt-3.5-turbo on GSM8K as reported in different sources. We acknowledge the variation in results across different papers. For example, according to chain-of-thought hub [3] (https://github.com/FranxYao/chain-of-thought-hub), the performance is 74.9 which is similar to our reported 74.2. But indeed, we find some papers reporting higher performance [4,5], which is about 80.8. We checked Opencompass [6] as well, they reported 78.2. All of these three sources have a much higher performance than our reported 74.2. This leads to an interesting question which motivates us to take a deep investigation.

First of all, we are very certain that the results we reported are not deliberately understated. We take a careful look into [4] who reported 80.8 and find they are using a different version of gpt-3.5-turbo. Our initial results were based on gpt-3.5-turbo-0613, while some of the higher performances reported in other studies utilized gpt-3.5-turbo-0301. We conduct experiments with gpt-3.5-turbo-0301 and surprisingly find that the baseline CoT is indeed above 80! We never expect such a situation although there are several papers observing and discussing similar phenomenons [7]. Therefore, we add the results with gpt-3.5-turbo-0301 as well to provide a more comprehensive evaluation. The results are shown below.

These results confirm a consistent improvement with our proposed Active-Prompt approach, irrespective of the GPT-3.5 version. We list the results with different versions in the same table for people to reproduce the results. We are grateful for your comment which helps us have a deeper understanding. In addition, the 0613 version is likely based on the continued training from 0301, which reveals a very interesting phenomenon — the forgetting issue of LLMs as training progresses. This has sparked our interest in conducting more in-depth research in the future. Thank you very much!

Q2. Ablation Study of Longer CoT Annotations

Thank you for suggesting an ablation study to differentiate the impacts of longer CoT annotations from our method. To explore this, we extended the length of the original CoT [1] annotations to an average of 155 words, comparable to our average length of 160 words. Here are the comparative results:

Our findings show that merely increasing the length of CoT annotations does not lead to improved performance, and in some cases, even reduces it. In contrast, our Active-Prompt method consistently demonstrates superior performance. This suggests that the selection of questions, rather than their length, contributes significantly to the improved results. Our approach effectively identifies and utilizes more informative examples for annotations.

Dear Reviewer DkmB,

We sincerely appreciate your insightful feedback once again. In response, we have diligently addressed each of your concerns and queries. Key updates include:

1. A deep analysis of the performance in Table 1. We explain that our reported results are not deliberately understated and confirm a consistent improvement with Active-Prompt.
2. Ablation Study of Longer CoT Annotations
3. Comparison and discussion with more baselines

Would you be able to spare some time to review our revised response and confirm if it adequately addresses your questions? We are eager to engage in further discussions and provide additional clarifications on any new queries you may have.

Thank you very much for your time and consideration.