Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning

Thank you for your insightful feedback, especially your recognition of our comprehensive evaluation.

Here are the responses to each of your questions:

W1/Q1: The weaker LLM may be overconfident in incorrect knowledge and fail to send the questions to the stronger LLM.

Thanks for your feedback! Yes, we acknowledge the inherent limitation of the LLM cascade method, wherein if the weaker LLM exhibits a high level of confidence in an incorrect factual answer, the pipeline would return that answer. Addressing this issue may necessitate additional measures such as external knowledge integration or human feedback, which is beyond the scope of this work. We have added this point in the limitation discussion of our revised paper and will consider it as an important future work.

However, we note that our method is still effective on factual reasoning tasks. That is again owing to the fact that using different prompt representations could trigger different reasoning paths, which often results in more trustworthy answers when the two representations agree with each other. While completely addressing the overconfidence issue of LLMs is beyond our scope, we note that our idea of Mixture of Thought can indeed mitigate this issue.

We have verified this idea in the newly added Appendix J based on the StrategyQA dataset. An example is shown in Figure 10. For the question "Is a curling iron necessary in curling?", the golden answer is "No, curling is an ice sport and doesn't need a curling iron". However, most of the CoT answers are "yes" with hallucinations about the concept of "curling". In contrast, most of the PoT answers are "No". The PoT processes typically list the necessary equipment for curling, such as "curling stone" and "broom", and then check if "curling iron" is on the list. By checking the consistency between CoT and PoT, MoT-1D-Vote is thus able to identify the incorrect or untrustworthy answer.

It is worth noting that, judging from the results of Figure 9, PoT is not better than CoT, but the combination of CoT and PoT generates diverse thoughts and answers, instead of leaning towards one kind of thinking, thus reducing errors in factual reasoning. Therefore, we can still leverage consistency checking across MoT prompts in decision-making to check if the answer from the weaker LLM is trustworthy in factual-based reasoning tasks. For more details, please refer to Appendix J.

We hope that this experiment can address your concern and once again demonstrate the effectiveness of our method on reasoning tasks. If you have further questions or comments, please let us know and we will try to address them during the remaining rebuttal period!

Q1: How exactly does the MoT-2D setting work?

We apologize for the confusion. For MoT-2D, we sample from two prompts, i.e., one prompt including M shots of examples written in CoT, and another prompt including another M shots of examples written in PoT. The verification score can then be calculated using the outcomes of these two prompts following Eq (3). To eliminate randomness caused by the pairing between demonstration examples and representations, in our experiments, we reported an average result of two cross-pairing. That is, we experimented with CoT1, CoT2, PoT1, and PoT2, where CoT1 and CoT2 denote prompts based on two sets (Set 1 and Set 2) of demonstration examples, but all written in CoT, respectively, and PoT1 and PoT2 similarly denote prompts based on Set 1 and Set 2 of demonstration examples but all written in PoT, respectively. The reported result is an average of pairing CoT1 with PoT2, and pairing CoT2 with PoT1.

Q2: If temperature 0.8 yields better results (Fig. 5), why is this not your default? Did you try to increase the temperature further?

No, we didn’t try to tune the temperature. The setting of T = 0.4 is from the well-known source paper (https://arxiv.org/abs/2211.12588). We borrowed their experience and intentionally kept the same temperature in our initial experiments. In the robustness analysis, although we found that we can get a higher accuracy with T = 0.8, re-running all experiments with a different temperature can consume a lot of money and time. Therefore, we have not repeated the experiments with a new temperature. We did not try to increase the temperature further, but this could be an interesting investigation in the future.

Q3: In Figure 4, what threshold was used to decide whether answers were consistent or not?

We apologize for the confusion when describing our Figure 4! The “consistency rate” mentioned in the Y-axis of Fig 4 refers to the same consistency or agreement score in our Eq (2). Below, we clarify how we collected the statistics in Fig 4, and these details have been clarified in our revised draft:

For each vote-based decision-making method, we first group questions into “easy” and “hard” based on whether the weaker LLM can answer them correctly (i.e., whether the majority-voted answer $A^w$ is correct or not). For each answer, we then calculate its consistency/agreement score following Eq (2). The Y-axis of Fig 4 reports an average consistency score across all easy (blue bar) or hard (green bar) questions.

Our results in Fig 4 imply that MoT-Vote outperforms CoT-Vote and PoT-Vote because it often assigns relatively lower consistency scores to hard questions while relatively higher ones to easy questions. As a result, when setting up the vote-based threshold, it is more successful in identifying hard questions (what weaker LLM cannot solve) and passing them to the stronger LLM, while saving costs by keeping the easy questions (what weaker LLM can solve).

Q4: Why do you think QA-based external verification with GPT-3.5 performed poorly? Does it incorrectly validate many incorrect answers as trustworthy? Have you tried increasing the temperature of the QA-based verifier, to understand the actual distribution the model places on "yes, trustworthy" vs. "no, not trustworthy"?

We gave our analysis at the end of section 3.5, that is, "It's an intrinsic challenge of deciding question difficulty and answer correctness solely based on their textual descriptions." Similar conclusions are also mentioned in some other papers (https:// arxiv.org/pdf/2303.17651.pdf, https://arxiv.org/abs/2306.13063).

From Figure 8, we could learn that the decision maker precision of the QA-based external verification is lower than our approach, indicating that many untrustworthy answers are trusted incorrectly (i.e., false positive). We increased the temperature in Appendix I and found that it could help the LLM-QA method but is still worse than our approach.

Thank you for recognizing our work being of high quality and clarity and for providing us with insightful suggestions!

We have revised our manuscript and addressed all the points you mentioned:

W1: In the vote-based methods, how calibrated is the distribution over sampled answers?

Thanks for your feedback! We agree with your suggestion that fine grained calibration analysis is important. In Section 3.3 and Figure 4, we have tried to have more in-depth analysis on the consistency or agreement rate of different vote-based approaches. In our updated Appendix I, we further include a calibration analysis as you suggested. In our analysis, we compare MoT-1D-Vote, CoT-1D-Vote, CoT-2D-Vote, with two variants of LLM-QA following the design of Kadavath et al. (2022), employing T=1 and T=2. Detailed results of this experiment are presented in Figure 8 (left).

Our analysis indicates that all decision-making methods yield a monotone calibration curve, implying that when they have higher confidence in a certain answer, the answer is generally more likely to be true. But there is no a significant difference among these approaches in terms of their calibration degree.

However, we wanted to note that achieving perfect calibration with $n/K$ as the confidence score is not necessary for our task. A more direct comparison of the accuracy of the subset satisfying $n/K$ greater than the confidence score among different decision-making approaches is in Figure 8(right). We observe the subset accuracy increases monotonically with a larger $n$ and our method is better than the LLM-QA method, which explains its superiority in the main experiments. For more details and discussions, please refer to Appendix I.

W2: The lack of transparency around OpenAI's pricing model.

We agree with your concerns regarding the lack of transparency around OpenAI's pricing model, which may make our results hard to interpret. In our experiments, we have assumed that the monetary cost difference between GPT-4 and GPT-3.5-turbo can reflect the difference in their computational costs, hoping that our results could still provide insights for the latter case. Transferring our results to the cost model of LLAMA2 could be tricky because LLAMA2 can have very different performance compared with GPTs (see Section 3.6 for our effort on this aspect).

To increase the transparency under this restricted condition, we instead decided to release all inputs and outputs from our experiments, as well as a Python script for calculating the token counts for each decision maker on the dataset. We uploaded our demo script with GSM8k dataset and will release the script for all datasets in the future. In this way, researchers or developers in the future could play with our token counts under any monetary or computational cost measurement they prefer. While this may not completely resolve your concern, we hope our effort can still contribute towards a more transparent use of our approaches.

Thank you for your valuable comments and the endorsing of our methods and experiments.

We have addressed the questions below. If you have further questions or comments on our response, please don’t hesitate to let us know!

Q1: Instead of just the answer as a hint, what if we give the entire CoT or PoT from one of the prompts as a hint? Will that help?

Thank you for your suggestion! Yes, how to better utilize the information from the weaker LLM in prediction is important. We conducted the experiment with the entire CoT and PoT in the GSM8k dataset as hints. In the previous setting, we use the sentence in the prompt: "Hints: The answer may be close to {CoT Answer} or {PoT Answer}". In the new experiment, we replace it with "Hint 1: {Entire CoT Process} Hint 2: {Entire PoT Process}". We conducted the experiment with the examples that CoT and PoT don't have the same answer. The performance for GPT-4 with the entire CoT and PoT is 0.851, which is lower than in the previous setting (0.867 in Table 12).

We have found that leveraging the entire CoT and PoT cannot yield an improvement in performance. Moreover, this approach incurs significant additional costs by necessitating the inclusion of the entire thought process in all demonstration examples, contradicting our primary objective of cost efficiency.

Q2: What if we ask multiple questions at once? won't we reduce the cost more? (2/3 Questions with context as prompt)

This is a good suggestion! To answer this question, we have followed the “batch prompting” setup of Cheng et al. (2023), where we grouped a batch of 4 test questions into each API call of the weaker LLM, in addition to the original 8-shot demonstrations. Like in our previous experiments, we obtain multiple samples from running the weaker LLM, and the verification-based method (Eq 3) can then be leveraged independently for each test question. If the answer is rejected by the decision maker, we then feed the rejected cases into the stronger LLM. We observed adding batch prompting can further reduce costs but slightly compromise accuracy. More details can be found in Appendix H of our updated draft.