GAOKAO-Eval: Does High Scores Truly Reflect Strong Capabilities in LLMs?

1. What does “human-like reasoning” mean? The term is used in several places but lacks a clear definition. More importantly, would “human-like reasoning” still be important if the LLM already achieves “human-like performance”? Addressing these questions could better motivate the research.
2. The performance of the new model is only marginally better than 4o (in “Science Total” and “Art Total”), even after being trained with an extensive GAOKAO-related corpus. What if 4o were fine-tuned on the same (or a subset of the) corpus? Additionally, what is the key message or finding conveyed by including the WQX model in the results? The necessity is unclear.
3. o1’s reasoning ability is mentioned and the finding looks promising; however, the internal reasoning process of o1 is opaque to users and the impact of CoT or other reasoning techniques on white-box models is not explored. Would CoT help reduce the inconsistency?

Minor:

1. line 23: "anomalous consistant performance across various question difficultiess" should be "consistent" and "difficulties".
2. line 25: "we find": "w" should be capitalized.

• The paper lacks clarity: o How are the human results conducted? What are the grading guidelines? How to distribute the tasks? How to validate the human evaluation process? o The paper uses Rasch model to simulate human performance. However, there lacks clarifications why GAOKAO performance could be simulated by Rasch model. The actual human performance distribution might be similar to the LLM’s. o Line 274 mentions the difficulty of questions. How is exactly the hybrid approach with human annotations and LLM scores? • The paper claims that o1’s reasoning-as-difficulties can mitigate the mismatch between the human performance and LLM’s performance on the benchmark. However, the paper lacks experiments on the performance distribution of o1 on the benchmark, and it is still unknown how this performance distribution aligns with the actual human performance distribution, which is also lacking in the paper. • The paper contains a few grammar errors and typos: Line 23: ‘consistant’, ‘difficultiess’, Line 26: ‘we’, ‘phenomenon’ should be plural, Line 459: ‘cabilities’, Line 527: ‘discoverys’, and more.
• The motivation of this paper is questionable. Previous benchmarks such as GAOKAO-MM and GAOKAO-Bench are proposed to evaluate the comprehensive capabilities of LLMs. However, this paper shows another point that human-aligned LLMs should have similar performance distribution as humans. Wouldn’t LLM research make better LLMs that have higher scores on tasks where humans perform poorly? Unlike improving safety and reducing toxicity through human-alignment, aligning human capability in reasoning tasks might not be a good idea.

• The Rasch model is commonly used in human testing. But it is unclear if the Rasch model is the best fit for modeling LLM behavior, especially without fully exploring/discussing alternative psychometric models.

• Some descriptions seem exaggerated. GAOKAO-Eval primarily assesses knowledge-based aspects of LLM performance, focusing on subject knowledge and question-answering within a constrained exam format. This scope limits its comprehensiveness as a benchmark for LLM capabilities, which is inconsistent with what is described in Section 2 Paragraph 1.

• The process of human involvement is not clear. The study involves 54 human graders without disclosing ethical considerations, which raises potential concerns.

I think the paper can be significantly improved and revised to clearly articulate the experiments, results, and insights.

1. The paper’s general message that LLMs’ performance varies across similar question types and that there is anomalous consistency across difficulty levels is well-studied in the literature. It would be beneficial if the authors focus on their dataset to showcase how models perform across different subjects and difficulty levels, highlighting what types of problems they perform well on versus those where they fail, and providing potential reasons why. Currently, results are aggregated to show performance variations across models on different difficulty levels.

2. I found it very difficult to interpret the results, as none of the figures provide a clear explanation of the experiment, insight, or key takeaway. For example, in Fig. 4, you show overall performance across models by subject, but do not clarify what the takeaway is from this figure. Does it imply that WQX and GPT-4o perform the best on this dataset? What is the overall accuracy on this dataset? It’s unclear what the models' performance is on the entire dataset.

3. Similarly, Fig. 5 lacks an explanation of how human ratings and LLM-based judgments were incorporated into ELO. The graph only shows the difficulty level for 11 questions. What does aligning difficulty level with expert judgments mean? Why are only GPT-4o results shown? What does the difficulty of extracted questions signify?

4. In Fig. 6, why is the IRT fit across all model results instead of fitting it at each model level to show, for example, whether GPT-4o outputs across difficulty levels align with human abilities? This result is unclear.

5. Fig. 7a has a grey area—what does this represent? How is difficulty determined by humans or models? The phrase “across models” is also unclear regarding what this graph is meant to demonstrate

6. In line 357, you mention, “our difficulty ratings are well-aligned with human perception and accurately reflect the human-aligned capabilities of LLMs.” How did you arrive at this conclusion?

7. What is the takeaway or insight from Fig. 8?

8. Where is eq 3 applied?

9. Figure 11 requires more detail. What does incorporating O1 tokens mean? O1 provides the steps and final answer but not the backend exploration or raw tokens, so what is meant by this?

10. The explanations and insights for Figures 11a, b, c, and d are poorly articulated.

11. Why not compare with other open-source multimodal models like LLavaOneVision and LLavaNext, which have shown to be more powerful on multimodal data.

12. How were the human raters selected? details of inter-rater agreement etc., should be provided

Apart from the above the key questions for me are:

1. Given that variants of GAOKAO-Bench and GAOKAO-MM already exist, what is the true novelty of this dataset? While the authors mention it is secure and non-leaky, the other two datasets are as well. What differentiates the construction of this dataset compared to the other two, establishing it as a key contribution?

2. If the novelty does not lie in the dataset itself, then the key contributions should focus on the insights derived from the data to deepen our understanding of LLM capabilities. Unfortunately, the paper does not fully address this aspect, as the authors primarily report aggregate numbers without clearly presenting key takeaways beyond the general message that LLM performance does not align with human abilities. I would like to see some key insights or takeaways derived from the experiments that are generalizable and hold broader significance for the community.