TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles

There are a few concerns I have for this benchmark

1. The task seems already quite saturated by decent LLMs. There is already a filtering for many easy to judge questions. "Ultimately, from the original 26,000 entries, we annotated 4,448 guesses. We conducted preliminary tests across all LLMs and filtered out simple questions that all models answered correctly. On the remaining 1,699 entries, ..." Even so, the accuracy on this benchmark can be around 80%, which means the unfiltered accuracy should be more than 90%. The remaining questions are also very likely to be ambiguous to judge. Given how easy this benchmark could be, I am concerned about the adoption of it.

2. The paper claims "Other dynamic evaluation methods based on strong models or manual efforts may introduce biases and incur high costs and time demands, hindering large-scale application." in Abstract and "Therefore, this paper proposes TurtleBench, an evaluation benchmark with a continuously updating dataset, offering concise and easily quantifiable evaluation results, ..." in Related Work. I don't see how the dataset can be continuously updated without (high) human annotation cost. Also once the benchmark is released, it can be found in the training corpus of LLMs which makes evaluation on future collected data appear to be easier for LLMs.

3. There is also usually some violent or horrific content associated with these lateral thinking problems. I am also slightly concerned about ethics.

• Experimental design: The prompt a model under evaluation receives instructs the model to answer correct or incorrect based only on the surface story, violating the premise of the benchmark and incentivizing the model to disregard the bottom story which should be the basis for logical coherence.
• Statistics: The authors claim that wrong responses from o1 have more tokens than correct responses. However, they lack a statistical test (such as significance testing under a mixed effects linear model) and using statistical heuristics for signifiance on their boxplot visualization in Figure 4 shows that this is false, as the confidence intervals overlap nearly 100%.
• Data quality: Figure 11 shows an example story "A Painting". The surface story includes irrelevant but seemingly important information for inferring the bottom story. In this case "my scalp tingled" is irrelevant to inferring the bottom story from a user's perspective, hence the data generation process may be contaminated by the difficulty of the task itself. This may also be by design to lead someone astray, but this seems to detract from the benchmark.
• Claims of objective results (minor): Similarly, in Figure 9, one of the user entries are "This soup tastes different from the human flesh he ate before" where the fact that "he ate [human flesh] before" is not in the surface story at all, suggesting that the user had separate information or they had memorized the story from a previous session. Additionally, in Figure 11, "I feel scared" is a somewhat subjective entry, though the story would imply this.
• Memorization: The authors claim that their benchmark "[reduces] the likelihood of models gaming fixed datasets for score inflation" through their interactive design. However, the stories themselves may be memorized, as evidenced above by the users themselves. The data is indeed new so this can be seen as solving this, but the stories themselves may be memorized.
• Dataset filtering: The authors filter for "something" to remove ambiguous entries, but there are many ways that "something" may be used in an unambiguous context.
• Interactivity: The authors claim it is an interactive benchmark but there is no interactivity from the model's side, only from the user side. The model's are queried in exactly the same way you would for a static benchmark.
• Related work: The related work section before the conclusion misses key papers introducing interactive benchmarks with objective resolution criteria, such as capture-the-flag challenges in Cybench and other papers.
• Conclusions: Due to the above issues, the conclusions of model performance are dubious. Additionally, the assumptions made about the design of o1 do not hold. The authors assume o1 is composed simply of CoT and excludes self-consistency functionality, however if you observe the performance difference between CoT prompting of GPT-4o and extrapolate to similar inference costs as o1, you see a significant and large performance gap between o1 and simple CoT, implying that o1 has more functionality embedded in its architecture than just CoT. I can provide sources for this statement and a series of papers that may be involved in the creation of o1.
• Translation: How the translation is done isn't expanded on but this may be a critical bottleneck to the English benchmark's data quality.

1. One of the main contributions of TurtleBench, as claimed by the authors, is its ability to address the problem of static evaluation benchmarks, which are prone to contamination and compromise the reliability of evaluation results. They argue that TurtleBench is dynamic because the data is collected from real user guesses from online platform. However, once the data is collected and filtered, it becomes static and no longer evolves, making it susceptible to contamination. The process of preparing TurtleBench data involves extensive manual effort for checking and filtering, resulting in a dataset that, once finalized, remains static.
2. Limited size and scope. TurtleBench ended up with only 32 stories, which is an unconvincing number. Additionally, it is unclear whether these 32 stories represent different categories, what specific content each story covers, or how they can comprehensively test the reasoning abilities of LLMs. The article does not address these concerns, so while the evaluation approach is quite interesting, there isn't enough evidence to support its comprehensiveness.
3. Limited analysis. The paper raised an interesting point: the OpenAI o1 series, which is well-known for its reasoning capabilities, performs worse than other LLMs on TurtleBench. While the authors suggest possible reasons, such as the o1 models focusing excessively on details, they do not explore these explanations further. Additionally, there is no convincing ablation study to support their hypotheses.

In their introduction, the authors criticize static benchmarks like MMLU for lack of logical reasoning, and contamination. However, it is unclear how the author’s proposed benchmark mitigates these problems. Mainly, despite crowdsourced user queries, the authors still must hand annotate each correctness label (which took 2 weeks according to line 66 in the main figure). It is doubtful that this process is continuously replicable, such that a new fresh set of labeled queries can be released in frequent intervals so that contamination is avoided. As such, it seems the author’s implementation is not substantially different that a regular static benchmark other than being based on lateral thinking puzzles and crowdsourcing queries (but not labels). Therefore, this benchmark still contains all the weaknesses the authors outlined for existing static benchmark implementations.

Moreover, the claims (lines 412-414, 418-419) of addressing the balance between real-worldness and verifiability seem unsupported. Lateral thinking puzzle judgments are an interesting aspect to test LLMs on, but are not real-world in themselves. Alternatively, the authors have not shown that excelling at this task is correlated to excelling at some other open-ended task that is much more applicable but hard to judge, e.g. general reasoning capability, or more human preferred. TurtleBench is based entirely on an LLM’s ability to judge queries on 32 different lateral thinking puzzles. To make the claims on real-world application supported, the paper should explore how this connects to performance on actual real-world scenarios in depth. For more details, please see the questions section for specifics.

Additionally, the section analyzing o1’s performance (section 3.4) needs more supporting experimentation. Additional questions and recommendations are in the questions section.

1. The paper lacks detailed information on how the user guesses were annotated for correctness.

2. Since the data comes from users interacting with the Turtle Soup Puzzle platform, there may be inherent biases based on the user demographics and the nature of the puzzles.

3. While accuracy and F1 scores are reported, the paper does not provide enough detail on how these metrics are calculated

4. The results are presented without statistical analysis to determine if the differences in model performances are significant.

5. The OpenAI o1 models could not be adjusted for temperature and top-p settings