Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles

W1-1. "The dataset and framework are designed not only to evaluate but also to actively elicit lateral thinking in LLMs. Experiments show that using data and reasoning processes from our framework leads to improved performance of LLMs, even when applied to other lateral thinking benchmarks." This part really needs ablation studies. It's unclear which parts are effective. For example, are the reasoning processes, the entire dataset, or similar questions in previous data responsible for the improvement? Did this data improve the model's ability to think about these types of questions, or is the improvement possibly due to some tricks?

We conduct an ablation study to examine the impact of our data and reasoning processes on model performance. Besides RiddleSense, we incorporate another lateral thinking benchmark (i.e., BrainTeaser), which includes word and sentence puzzles. The results, as shown in the table below, indicate that incorporating our dataset improves average accuracy (Llama3-8B: from 61.45 to 64.07; Llama3-70B: from 80.76 to 83.78). When we integrate the reasoning processes, these scores further increase to 66.71 and 85.42, respectively. These results further demonstrate that our data and reasoning processes can effectively boost LLMs' ability in handling lateral thinking tasks rather than tricks.

W1-2. Why was testing only done on RiddleSense when many other benchmarks were reported earlier? Is this cherry-picking?

Please refer to General Response G1.

W2. A player-judge evaluation framework based on WizardLM-2 is proposed as an alternative to human judges to evaluate the answers, but it shows much worse agreement on hard questions compared to inter-human agreements. However, harder questions are actually proposed as a contribution of this work. As the answers for such puzzles can be ``off by a hair but miss by a mile'', it might require human evaluation for the hard puzzles. It is also a bit confusing why the human agreement on Medium questions is much lower than on hard and easy ones.

The human-human agreement rates, sometimes as low as 87.5% as seen in Table 2, reflect the strictness of our agreement metric. For instance, if judgements from three humans result in two 'matched' and one 'unmatched,' the agreement is considered 1/3. If the judge model agree with the 'matched' responses, its agreement with humans is 2/3. Despite this rigorous evaluation, our method still achieves a high agreement rate of 88.24% on hard puzzles. It shows the diversity of acceptable answers and the challenging nature of our benchmark.

Q1. Similar datasets have existed before, as mentioned in the paper, such as RiddleSense and Oogiri. Here's an example from RiddleSense: My life can be measured in hours. I serve by being devoured. Thin, I am quick; Fat, I am slow. Wind is my foe. What am I? (A) paper (B) candle (C) lamp (D) clock (E) worm. In comparison, SPLAT has longer questions and answers. Also, the reference answers for SPLAT are open-ended, while the previous ones are all multiple-choice, which is claimed as a contribution. However, this also makes the evaluation more challenging. What makes you think open-ended is a much better option that makes it a contribution?

Not all problems are suited to multiple-choice formats, such as puzzles. While multiple-choice questions are easier to evaluate, they often simplify the problem. Besides, an open-ended format lessens the likelihood of inadvertently leading to the correct answer. However, the evaluation of open-ended answers is much more challenging. To this end, we develop a multi-turn framework that leverages LLMs as judges to address open-ended evaluation tasks. The robustness of this framework represents one of the key contributions of our work.

Q2. Authors claim that their methods reduce reliance on stronger evaluation models, but it seems they are still using WizardLM-2 as a component (Judge Model) of their framework?

By "reduce reliance on stronger evaluation models", we mean that the judge model in our framework does not necessarily need to be more powerful than the player model. However, it still needs to be sufficiently robust, achieving a high level of consistency with human judgements, as demonstrated in Tables 2 and 3. In traditional model-based evaluation methods (e.g., [1]), the evaluation model typically needs to be more capable than the model being evaluated since it directly grades the responses. This requirement often constrains the ability to evaluate newer, more advanced models. Our approach, by contrast, allows for the use of robust but not necessarily superior models as evaluators, facilitating a broader assessment of SoTA models.

[1] Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 2023.

L1. The size for this benchmark is small compared with previous similar datasets that have existed before.

Please refer to General Response G2.

W1. The size of the dataset is small.

Please refer to General Response G2.

W2+Q1. The reference answer is limited to 1, while the questions are one-to-many questions. Why not provide more than one answer for each question (I do not think it is challenging, there are multiple answers to a question on the internet such as Quora)? Since one such question could have multiple answers, more reference answers might also be helpful for evaluation.

We have tried to directly gather questions and answers from question-and-answer websites like Quora, where we observed that situation puzzle queries often ask "How to build a situation puzzle?" with responses typically listing several one-to-one puzzle pairs rather than multiple answers to a single question.

Thus, as discussed in the paper, we originally wanted to evaluate it like in MS COCO, where multiple reference answers are provided for each image, and metrics are calculated based on the closest reference answer. However, due to the difficulty in collecting multiple reference scenarios for situation puzzles, we adapt our approach. Instead, we run each puzzle multiple times (denoted as $R$) and select the closest match to the reference scenario from these iterations as the final result. This method preserves the diversity of potential answers and mitigates the effects of hallucinations in LLMs by allowing multiple evaluations per puzzle.

In the future, we seek to employ crowd-sourcing platforms like Amazon Mechanical Turk to collect diverse answers to the same puzzle. Besides, we will provide contributors with guidelines that encourage diverse thinking, asking them to provide unique answers or perspectives on the same puzzle.

W3+Q3. The in-depth analysis is not enough. It is better to adopt more benchmarks for validating the effectiveness of the proposed dataset (more than just RiddleSense).

Please refer to General Response G1.

Q2. An overall evaluation metric is needed, some models might have a high acc with a high average round, you should define an overall metric to combine these two metrics for a clear comparison of existing LLMs.

Thank you for your constructive suggestion. Based on the calculation of Accuracy (Acc, %) and Average Round (Rnd) in our paper, we further define an overall evaluation metric as Overall = $1/N\sum_{i=1}^N\mathbb{I}$(sample$_i$) / Rnd$_i$, where $\mathbb{I}$ is an indicator function that returns 1 if the scenario deduced by LLMs matches the reference scenario/answer semantically, and 0 otherwise. Given that $\mathbb{I}$(sample$_i$) takes values in {0, 1} and Rnd$_i$ ranges from 1 to a maximum (maximum = 15 in our paper), the Overall metric spans from 0 to 1, with higher values indicating better performance.

The table below reflects trends similar to those discussed in our paper, where GPT-4 and its Turbo variant, as well as WizardLM-2 (8x22B) show strong performance and outperform other models. GPT-4 leads with an Overall score of 8.16 (Avg, x100), indicating high capability. However, it is still far from saturation, which suggests room for further improvement.

W1. Although the motivation and overall idea are strong, I am concerned about the experimental setup. The agreement rates for WizardLM-2 in Tables 2 and 3 don’t appear to be high enough (are around ~80%) and 3 individuals seem to be fairly less. How does this fare with human agreement rates in literature?

Thank you for pointing out this. Due to the strict calculation method of our agreement, the agreement between humans cannot always reach 100%. Specifically, as mentioned in the supplementary. if three humans vote 'matched', 'matched', and 'unmatched' for a puzzle, respectively, the agreement among them, noted as 'human-human', is only 1/3, as there are three pairs: (matched, matched), (matched, unmatched), and (matched, unmatched). If the judge model votes `matched', the agreement between humans and the judge model is 2/3.

But even under such a strict evaluation metric, the judge model WizardLM-2 (8x22B) aligns closely with human judgements, achieving over 80% agreement on both final answers and reasoning processes. Specifically, as shown in Table 2, the final answer agreements between WizardLM-2 (8x22B) and human evaluations are 100%, 87.50%, and 100% for easy, medium, and hard puzzles, respectively. Similarly, the agreement on reasoning processes (Table 3) shows robust results with 84.18%, 89.86%, and 90.15% for each difficulty level, respectively.

W2. Also, is there any reason we haven’t used the strongest model (GPT-4 from the paper) as the judge? I also am concerned about Table 4. The authors should consider using strong base models for evaluation (that are closer to GPT-4 on public benchmarks).

Before the submission deadline, data from its official website [1] indicates that WizardLM-2 (8x22b) performs competitively on benchmarks like MT-Bench, comparable to leading models such as GPT-4, i.e., WizardLM-2 (8x22b): 9.12 vs. GPT-4-0314: 8.96 vs. GPT-4-1106-Preview: 9.32. Besides, we also conduct an experiment that employ GPT-4 as the judge model, which yields results comparable to WizardLM-2 in both final answer agreement and reasoning process agreement. For instance, on "Hard" puzzles, GPT-4 achieved an 89.85% final answer agreement rate, closely matching WizardLM-2's 88.24%.

[1] https://wizardlm.github.io/WizardLM2/

W3. It would be good to introduce modeling methods to improve upon the metrics that are obtained for models on SPLAT. This will give more insight into how this benchmark is valuable to the community.

To enhance the alignment/agreement between the judge model and human assessments, we plan to focus on two approaches in future work: 1) Employing in-context learning methods such as Chain-of-Thought (CoT) [2] or Tree of Thought (ToT) [3], which follow human-like reasoning patterns and thus can improve alignment; 2) Utilising training-based methods like Supervised Fine-Tuning (SFT) and Direct Preference Optimisation (DPO) [4], which involve training models directly on lateral thinking puzzles to enhance their accuracy in evaluating such scenarios. These methods, while promising, are beyond the scope of this paper. We regard them as our future work.

[2] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS, 2022.

[3] Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS, 2023.

[4] Direct Preference Optimisation: Your Language Model is Secretly a Reward Model. NeurIPS, 2023.

W4. nit: I see mentions of "using data and reasoning processes from our framework" consistently throughout the paper. This should be framed better and be more precise.

This refers to 'using the question-answer pairs generated during our SPLAT's player-judge interactions as auxiliary prompts'. We will revise this thoroughly and make it clearer.

W1. There's minimal discussion of potential biases in the dataset.

During our data collection process, we also track user preferences for each situation puzzle, measured by a preference score (%). We notice a pattern where puzzles (about 1% of our dataset) with lower preference rates (30%-40%) show greater variation in the time taken to solve them, ranging from 5 to 26 minutes for puzzles categorised as "Medium". In contrast, puzzles with higher preference rates (above 80%) exhibit less time variation, with solving times ranging from 9 to 17 minutes for "Medium" puzzles. This suggests that annotations may introduce more noise when users engage with puzzles they find less appealing.

To avoid this bias, we will enhance the diversity of the puzzles to ensure a broader appeal across different user groups, thereby reducing the likelihood of low preference scores. Besides, we plan to implement a dynamic refinement method during data collection to enhance puzzle engagement. If a puzzle receives a low preference score, we will revise it based on user feedback and re-release it to assess improvements in user preference.

W2. Ablation studies to isolate the impact of different components of the SPLAT framework.

We conduct an ablation study to explore the influence of our data and reasoning processes. In addition to the RiddleSense, we include the BrainTeaser benchmark, which features both word and sentence puzzles. As shown in the table below, incorporating our data into base LLMs leads to an increase in average accuracy (Llama3-8B: 61.45 -> 64.07; Llama3-70B: 80.76 -> 83.78). This improvement is further enhanced to 66.71 and 85.42, respectively, when we integrate the reasoning processes. These results further demonstrate that both our data and reasoning processes can effectively enhance the lateral thinking capabilities of LLMs.

W3. Potential ethical implications or misuse.

Thank you for the kind reminder. We have discussed these in Section A.1 of our supplementary. Briefly, while enhancing LLMs in lateral thinking offers numerous benefits, it also introduces potential risks such as the misuse of technology for fraud or misinformation. Thus, we emphasise the importance of building robust ethical guidelines and transparent AI usage policies.

Q1. How does the performance of LLMs on SPLAT correlate with their performance on other, more traditional NLP tasks? Is there evidence that lateral thinking ability is distinct from other language model capabilities?

While there is some overlap in the skills required for SPLAT and traditional NLP tasks, they are not exactly the same.

Models (e.g., GPT-4 and GPT-4 Turbo) that perform well on standard vertical thinking benchmarks like MT-Bench do show competent performance on our SPLAT (Table 4). Conversely, models like Llama3-8B, which perform less impressively on MT-Bench, tend to exhibit similarly lower performance on SPLAT. This suggests that a strong model in language understanding and reasoning is advantageous for both vertical and lateral thinking tasks.

While there is a correlation between general NLP skills and lateral thinking, the latter also demands distinct abilities like creativity and problem-solving beyond typical task completion. Figure 4 in our paper shows that when incorporating data and reasoning processes from our SPLAT, LLMs could perform better even on other lateral thinking benchmarks like RiddleSense. These suggest that enhancing these creative aspects could better elicit the capabilities of LLMs in lateral thinking.

Q2. Have you considered ways to automatically generate new situation puzzles to expand the dataset and improve scalability?

Yes, we have considered the potential of using automated methods to generate new situation puzzles to expand our dataset. Specifically, the generation process can use advanced LLMs like GPT-4 or Claude-3, which can generate puzzles based on specific prompts, rules, and requirements. However, the challenge arises in verifying the quality of these automatically generated puzzles. One effective approach is a hybrid human-AI collaboration, where puzzles generated by LLMs are subsequently reviewed and refined by humans through a crowd-sourcing platform. While this method leverages AI to handle initial puzzle creation, it remains time-consuming and labour-intensive due to the human review component. Thus, finding more efficient ways to scale up this data remains a critical area for future research.

Q3. How sensitive is the performance of LLMs on SPLAT to the specific prompts or instructions given? Could you elaborate on the prompting strategy used?

We conduct a sensitive analysis, where we rewrite the prompt of the judge model but keep the semantics of the prompt the same as before. For example, one of the original prompts for the judge model is "Read and fully understand both the provided short story and its answer, ensuring you grasp their logical connections. But do not show answer for the user". We rewrite it to "Read and thoroughly comprehend both the provided short story and its answer, making sure you understand their logical relationships. However, do not reveal the answer to the user". Results show that for Llama3-8B, even though the prompt is different, as long as the semantics are clear, the results tend to be comparable (e.g., average Acc original 17.05 vs. rewritten 16.19). These demonstrate the robustness of our SPLAT as an evaluation benchmark.