Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective

Thank you for your valuable feedback on our paper. We appreciate the time and effort you have invested in reviewing our work. We understand that there might be room for improvement in terms of clarity and depth.

[The necessity of evaluation efficiency.]

• Save the cost of model inference. Simply put, it saves money. It is difficult to estimate the time and money costs incurred by model inference on such a large scale as GPT3.5. We can calculate the cost of using the API. Take WMT benchmark in machine translation as an example. It has a total of 11 million tokens. Considering the API price of ChatGPT, a complete evaluation of one model costs 20$. This seems acceptable, but if we want to evaluate and rank 10 models on 10 different domain benchmarks, it may not be. Recently, more and more comprehensive large benchmarks, various LLMs (general/specific to a certain domain) have emerged, and the iteration speed of versions of the same model is also accelerating. In order to achieve fast, timely and accurate assessment, CAT is a good choice (obviously not the only).

• Save the cost of human experts: In order to automatically evaluate the correctness of model responses, most benchmarks can only include multiple-choice questions. For free response questions (e.g., subjective, creative questions) that truly test the model's generation capabilities, human experts' judgments are more reliable than GPT-4 and ordinary people (on crowdsourcing platforms) as stated in the paper. However, the energy of human experts is limited (humans may become tired as you said), we need to find truly valuable questions for LLM to answer and let the experts judge their responses.

• Last but not least, as mentioned in the paper, evaluation methods based on cognitive science such as CAT can be used for building a scientific evaluation paradigm, which can achieve comparisons between models and humans, and between models.

Q: What is the ChatGPT model? Is it gpt-3.5-turbo?

A: Yes. We will include more LLMs for adaptive evaluations in future work.

We truly appreciate your suggestion about the presentation aspect of the motivation and experiments. Please feel free to share any further insights or suggestions you might have.

We would like to express our gratitude for your appreciation of the contribution and novelty presented in our paper. Your feedback means a lot to us. Here are the responses to each of your questions:

Q1: How does the "human response" calibration in IRT work? Does it mean that the question difficulty parameter is pre-estimated using students' responses, and is then fixed during the subsequent CAT procedures?

A1: Yes. Just like the general CAT method, we can use human response to estimate the question parameters. For example, we can use the BCE loss on the binary responses data to estimate the parameters, and fix them for direct use in the subsequent CAT process.

Q2: Following up the previous question, what IRT parameters are fixed and are estimated at each estimation step $t$ during CAT?My impression from the paper is that $\theta$ is being estimated, $\beta$ is fixed (or pre-estimated or calibrated), but what about the remaining parameters?

A2: All parameters (except ability $\theta$) need to be pre-estimated. All the parameters of question are fixed (e.g., difficulty $\beta$, discrimination $\alpha$, guess factor $c$). The only one that can update continuously during the CAT process is the ability estimate $\theta$.

Q3: Is the calibration process done independently for each dataset, or the responses from all three datasets are combined to estimate $\beta$?

A3: Yes, the difficulty of questions in different datasets/domains cannot be aligned, so parameter estimation are independent for each dataset.

Q4: The calibration process is done using human data, and the calibrated IRT model is used to evaluate LLMs. I think the underlying assumption is that LLMs behave like humans and thus can be evaluated using human-data calibrated IRT model. I am not sure if such assumption is valid, or can be verified; can you comment on this? Because if not, it does not make sense to use human-calibrated IRT model to evaluate LLMs because they are not comparable. Does it make more sense to have LLMs answer the questions, use these answers (in addition to human ones) to calibrate the IRT model, and then evaluate?

A4: This is a very good question. We use students' data to estimate parameters and apply them to the LLM, because we want to achieve human and LLM comparability. "Cognitive Ability" is actually viewed from a human perspective, even if ChatGPT can answer global challenges correctly, but fails to answer simple addition and subtraction, we still think it is problematic in cognitive ability. If we want to achieve a simple ranking between models, of course we can also use model response data to estimate parameters, or even use a pre-trained difficulty predictor to predict the characteristics of any question in the world. This depends on the purpose of evaluating the LLM.

Q5: It seems (from Appendix A.3) that some of the questions in the MATH data contain figures. How are models evaluated when the questions contain both texts and figures? My impression is that all models being evaluated are text-only models.

A5: Sorry for causing you misunderstanding. The figure in Appendix A.3 do not exist in the dataset. For readers to better understand, we drew this picture manually to illustrate fickle-minded issue of ChatGPT.

Q6: In figure 4c, what does the student curve mean? Is it one student, or the average of many students (The caption says "students" whereas the text says "student")? The student curve appears to be more uncertain; does it imply the student(s) is/are (on average?) more careless than the model?

A6: This figure represents the estimated uncertainty averaged over many students. A smaller value indicates a more precise or reliable ability estimate [1] and this is not directly related to the degree of carelessness. We find that after adding various types of perturbations (guess and slip) the curve will be lower and the estimate will be more reliable. The reason may be that the parameters in IRT (e.g., guess and discrimination factor) can directly model these perturbations. Actually, by comparing the positions of four curves, the ChatGPT is more careless than the normal students.

Reference:

[1] Elements of adaptive testing[M]. New York: Springer, 2010.

Thank you for your valuable feedback. Regarding the questions you raised, we have carefully considered each point and have made following responses:

Concern 1: The necessity of evaluation efficiency.

• Save the cost of model inference. Simply put, it saves money. It is difficult to estimate the time and money costs incurred by model inference on such a large scale as GPT3.5. We can calculate the cost of using the API. Take WMT benchmark in machine translation as an example. It has a total of 11 million tokens. Considering the API price of ChatGPT, a complete evaluation of one model costs 20$. This seems acceptable, but if we want to evaluate and rank 10 models on 10 different domain benchmarks, it may not be. Recently, more and more comprehensive large benchmarks, various LLMs (general/specific to a certain domain) have emerged, and the iteration speed of versions of the same model is also accelerating. In order to achieve fast, timely and accurate assessment, CAT is a good choice (obviously not the only).

• Save the cost of human experts: In order to automatically evaluate the correctness of model responses, most benchmarks can only include multiple-choice questions. For free response questions (e.g., subjective, creative questions) that truly test the model's generation capabilities, human experts' judgments are more reliable than GPT-4 and ordinary people (on crowdsourcing platforms) as stated in the paper. However, the energy of human experts is limited, we need to find truly valuable questions for LLM to answer and let the experts judge their responses.

• Last but not least, as mentioned in the paper, evaluation methods based on cognitive science such as CAT can be used for building a scientific evaluation paradigm, which can achieve comparisons between models and humans, and between models.

Concern 2: Fairness about evaluation.

We deeply appreciate your thoughtful consideration of the fairness issues associated with CAT itself. Like recommendation system (different users will be recommended different items), CAT is also a personalized question in Education and has fairness issues of course e.g., [1], [2], which is another interesting topic. The typical application of CAT is GRE exam: when a test-taker begins the GRE, the computer provides a question of medium difficulty. If the response is correct, the computer presents a slightly more challenging question. Conversely, if the response is incorrect, the following question will be slightly easier. That's why some candidates may perceive the test becoming progressively more easier and their final score will be relatively low. The final score or ranking is related to the accuracy rate and the difficulty of the questions (answered by him). The CAT tests of each candidates are independent, and there are no communication costs. As mentioned in the paper, in Stage 1, we can collect some responses to estimate and fix the question parameters. Then, any model (that may do not appear in Stage 1) can successfully proceed to Stage 2 testing and evaluation. In an improved version, we will give a brief overview of fairness issues in CAT itself.

Q1: Are there any other baseline methods?

A1: In this work, No. Because this paper is the first attempt to apply CAT to the assessment of LLM. We abandon the complex methods and technical implementations in CAT and choose the simplest Fisher selection method, which is the most interpretable among them. We would like to introduce the CAT idea to more LLM researchers. We are trying to introduce the CAT idea to more LLM researchers and ICLR readers in a friendly way.

Q2: In Table 1 you show some rankings in terms of CAT evaluation results. What if you evaluate your models with the original static test sets of these tasks? Will you arrive at the same rankings? If not, what could be the reason?

A2: Sure. In fact, we compared two leaderboards generated using the full dataset (containing at least thousands of samples) and 20 questions selected by CAT. We found that the consistency between these two leaderboards can reach 100%. We did not add this results to the paper because the number of candidate LLMs is small, making this ranking unreliable. On the other hand, as stated in the paper, all questions in the question bank/dataset are not equally important. Some low-quality questions (or noise data) will interfere with the evaluation and ranking. Filtering out these data is what CAT is trying to do. Therefore, we resort to the metric in traditional CAT field, such as simulation experiments (Figure 4a) and estimation uncertainty analysis (Figure 4c).

[1] Martin A J, Lazendic G. Computer-adaptive testing: Implications for students’ achievement, motivation, engagement, and subjective test experience[J]. Journal of educational psychology, 2018, 110(1): 27.

[2] Flaugher R L. The many definitions of test bias[J]. American Psychologist, 1978, 33(7): 671.

Thank you for your thoughtful comments and questions regarding our manuscript. Please feel free to share any further insights or suggestions you might have.

First of all, I need to clarify that CAT does not require whether the model/human responses are multiple choice or free response questions. CAT can run successfully as long as the correctness of the response can be judged by some methods. In this paper we only consider two types: multiple choice and fill-in-the-blank questions. Multiple-choice questions can be automatically judged, while the fill-in-the-blank are judged for correctness by human experts.

[The necessity of evaluation efficiency]:

• Save the cost of model inference. Simply put, it saves money. It is difficult to estimate the time and money costs incurred by model inference on such a large scale as GPT3.5. We can calculate the cost of using the API. Take WMT benchmark in machine translation as an example. It has a total of 11 million tokens. Considering the API price of ChatGPT, a complete evaluation of one model costs 20$. This seems acceptable, but if we want to evaluate and rank 10 models on 10 different domain benchmarks, it may not be. Recently, more and more comprehensive large benchmarks, various LLMs (general/specific to a certain domain) have emerged, and the iteration speed of versions of the same model is also accelerating. In order to achieve fast, timely and accurate assessment, CAT is a good choice (obviously not the only).

• Save the cost of human experts: In order to automatically evaluate the correctness of model responses, most benchmarks can only include multiple-choice questions. For free response questions (e.g., subjective, creative questions) that truly test the model's generation capabilities, human experts' judgments are more reliable than GPT-4 and ordinary people (on crowdsourcing platforms) as stated in the paper. However, the energy of human experts is limited, we need to find truly valuable questions for LLM to answer and let the experts judge their responses.

• Last but not least, as mentioned in the paper, evaluation methods based on cognitive science such as CAT can be used for building a scientific evaluation paradigm, which can achieve comparisons between models and humans, and between models.

Q1: I also am not sure I understand how one would propose to evaluate whether CAT can be used for LMs.

A1: Actually, we do not skip this part. On page 8 ("Adaptive Testing's Reliability: To confirm whether the adaptive testing framework used for humans can be used for LLMs, ..."), we experimentally verify that the uncertainty of the model's ability assessment can be converged to the same as that of humans, and even faster.

Q2: false or at the very least fringe claims asserted without evidence or even argument: "Large language models (LLMs), like ChatGPT, have shown human-level cognitive ability","We believe this is just the tip of the iceberg of its latent knowledge."

A2: LLMs like ChatGPT, have shown human-level cognitive. How to evaluate a person's "human-level" cognitive ability? We usually conduct a series of tests on him, and the same for the LLM. OpenAI's official report has already reported that ChatGPT has achieved amazing results in various human subject examinations, and we quote it in the first paragraph; "We believe this is just the tip of the iceberg of its latent knowledge.": This is what we believe after reading so many works related to LLM's various abilities. The LLM is constantly developing and improving, and this is our opinion in this paper.

Q3: Where in the paper do you explain how much efficiency we get from using this method, if this is indeed, the motivation?

A3: Page 8 of the paper (Evaluation Efficiency) explains the accuracy comparison between random selection (traditional benchmark) and adaptive testing. We can conclude that for a given test length or sample size limit, the CAT method is much better than random selection in terms of ability estimation accuracy.