TestAgent: An Adaptive and Intelligent Expert for Human Assessment

Thank you for your affirmation and feedback on our article. It seems that there might have been a mix-up in your review comments between our paper and paper ID 6485. Could you please synchronize the feedback for us? Here, we provide a detailed response to the review comments mentioned in the review of paper ID 6485.

Q1: In the final step of the framework, expert analysis is still required. This feels a bit contradictory to the goal of improving automation. Since you're already using LLMs, why not go a step further and design an evaluation agent to replace or supplement the expert analysis? This could boost efficiency and reduce dependence on experts.

We use expert analysis here to reduce diagnosis hallucinations of large language model and enhance the interpretability of diagnosis reports. If a agent is designed to replace diagnosis reports without any fine-tuning, it cannot guarantee the professionalism of the diagnosis reports generated by this LLM. Similarly, using an agent as an expert analysis can sometimes result in very neutral statements, making it difficult for testers to obtain useful information from such reports. So it is important to employ expert analysis supplemented with fine-tuning to make the results more convincing and reduce hallucinations of large language models.

Q2: While the framework diagram is clear, I think it could be made even more detailed.

We provided a specific example. Below is the complete process of implementing the MBTI test to facilitate better understanding.

Universal Data Infrastructure

TestAgent Planning

Report Generation

The following is your correct review comments. This is from https://openreview.net/forum?id=UVaPEthRKx&noteId=9KJyDYslMg

Summary: As an expert in large language models (LLMs), I must admit I’m not very familiar with cognitive diagnosis. So, my review will mostly focus on the LLM aspects. This paper introduces TestAgent, an adaptive testing agent based on LLMs, and honestly, the system has great potential. It conducts adaptive testing through interactive dialogues, which I think is a very cool idea. It helps solve some of the issues in traditional adaptive testing, like rigid question formats, noise in the data affecting accuracy, and overly simple output. By combining adaptive question selection and cognitive diagnosis, TestAgent dynamically selects personalized questions based on performance, identifies anomalies, and provides detailed diagnostic reports. What's even better is that it reduced the number of test questions by about 20% in several domains, such as psychological and educational testing, significantly improving testing efficiency.

Soundness: 4: excellent Presentation: 3: good Contribution: 3: good Strengths: Originality: What impressed me the most is the combination of LLMs with adaptive testing. This is a fresh approach that breaks away from traditional, static testing, making the process much more flexible and interactive. The ability to generate personalized questions using LLMs really improves the overall testing experience.

Quality: From the experimental results, TestAgent shows excellent performance in multiple fields. Reducing the number of test questions by 20% without sacrificing accuracy is incredibly valuable in real-world applications, saving time and maintaining reliability.

Clarity: The paper is well-structured, and I particularly appreciated Figure 2, which clearly shows how the entire system works. Even readers who aren’t familiar with this area can easily understand the role of each module.

Significance: In terms of practical applications, TestAgent has a lot of potential. Whether it's in education or mental health, personalized assessments and real-time feedback are crucial. A system that can dynamically adapt to user needs could have a big impact.

Weaknesses: In the final step of the framework, expert analysis is still required. This feels a bit contradictory to the goal of improving automation. Since you're already using LLMs, why not go a step further and design an evaluation agent to replace or supplement the expert analysis? This could boost efficiency and reduce dependence on experts.

While the framework diagram is clear, I think it could be made even more detailed. For example, showing the flow of data between different modules could make it easier to understand how the system operates, especially for readers unfamiliar with the field.

The paper discusses two key components of TestAgent: adaptive question selection and cognitive diagnosis. I think an ablation study would be helpful to evaluate the individual contributions of these components. It could also be interesting to see what happens if human experts take over one of these components, providing insight into human-machine collaboration.

The appendix gives some examples of the tests, but it doesn’t show the full prompts sent to the agent. For an LLM-based system, the design of the prompts is crucial, as it defines the task and the agent's role. I suggest the authors provide more detail about how these prompts are structured to give a clearer picture of how the system operates.

Questions: In the final step of expert analysis, could an evaluation agent replace the expert? If so, how would this change the system’s automation and efficiency? Has this alternative been considered?

Could you provide more detail about the prompts used to define the agent’s tasks? How do these prompts affect the agent’s behavior, especially in more complex testing scenarios?

Have you considered conducting an ablation study to test the individual contributions of the adaptive question selection and cognitive diagnosis modules? How would the system perform if one of these components was replaced by expert intervention?

Flag For Ethics Review: No ethics review needed. Details Of Ethics Concerns: No ethics review needed.

Rating: 6: marginally above the acceptance threshold Confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked. Code Of Conduct: Yes

We would like to express our sincere gratitude for taking the time to review our paper. Your insights are valuable to us. Below, we provide our responses to the points you raised."

Q1: What exactly is being measured through human assessments is unclear throughout Section 2, i.e. what is exactly captured by θ and what are concrete examples as defined by the datasets considered in the experimental section?

Psychometrics typically represents a test-taker's ability as a vector $\theta$. In Item Response Theory (IRT), $\theta$ also represents the test-taker's ability, where a higher value indicates stronger proficiency. This is explained in more detail in our article. For example, in the MATH dataset, which is a student math test, $\theta$ measures the student's ability level. The higher the value of $\theta$, the stronger the ability, and the higher the probability of answering questions correctly. In the MBTI dataset, $\theta$ is represented as a four-dimensional vector. For instance, in the (I/E) dimension, the larger the value of $\theta$ in this dimension, the more extroverted (E) the individual is.

Q2: It was challenging to understand how each component of the TestAgent framework depicted in Figure 2 was studied in the evaluation. It seemed like there were many moving parts and was unclear whether the entire framework was being evaluated or just components.

It seems there has been some misunderstanding. In Section 2, we generally introduced the purpose of each component, and the components work in collaboration, not as isolated moving parts. The representation in Figure 2 is intended to clearly show the role of each module, rather than suggesting that each module functions independently. Our paper does not analyze from the perspective of individual components; rather, it focuses on the entire TestAgent framework. Therefore, in the final experimental section, it is not possible to evaluate the components separately, as the evaluation requires their joint collaboration to achieve the intelligent testing of TestAgent.

The reviewer expressed interest in some of the training details of our paper, and we provide the following explanation.

Ability Classifier Training

Cognitive diagnostic models provide a vector $\theta$ as the diagnostic result; however, this is not interpretable. For the vector $\theta$, the MBTI test includes four dimensions: (I/E), (N/S), (T/F), and (J/P). Therefore, we train a classifier where the input is the diagnostic model's $\theta$, and the output is a four-dimensional vector corresponding to these four dimensions, thus transforming the abstract diagnostic number into features.

In specific terms, for a personality classification data labeled as $Y_{label}$, cognitive diagnostics provide a diagnostic result $\theta$ based on response to questions. Let $f$ be a mapping function that can map personality classifications to a 0-1 vector, for example, $f('ENFJ') = [1, 0, 1, 0]$. Let $g$ be the classifier we aim to train. The loss function can then be written as:

$L(\theta) = CrossEntropy Loss( g(\theta) , f(Y_{label} ) )$. With this, the classifier training can be implemented.

Performance Test Details

We repeated the experiment several times for each baseline. We used t-tests to assess the performance differences between different methods. Specifically, for each pairwise comparison of methods, we conducted independent samples t-tests under the assumption of equal variances. We tested whether the differences in the ACC and AUC metrics were statistically significant. For results with a p-value ≤ 0.01, we considered the difference to be statistically significant and marked it in bold in the table. The bolded text you mentioned is due to the significant advantage of TestAgent+KLI over the other baseline models. Although the AUC@50 values are identical between TestAgent+KLI and MAAT, the best result for TestAgent+KLI is 67.00, while the best result for MAAT is 66.60. Therefore, we have highlighted this difference using bold text.

Thank you for your thoughtful review and the recognition of our paper. We will address each of your questions below.

Q1: Question: How does the model perform when evaluated on real-world data? Are there specific reasons for relying solely on synthetic data in the experiments?

Although our MBTI and SCL datasets are synthetic, the MATH dataset is a real dataset, obtained from a private company. Detailed information about the dataset is shown in the table. For synthetic datasets, if there is a corresponding real-world dataset available, using the real dataset is undoubtedly the best option. However, unfortunately, it is very difficult to access interaction data between questions and test-takers for every domain, as this information is relatively sensitive, especially in areas like mental health and personality testing. Our framework aims to propose a universal method for data construction and cognitive diagnosis model training that can be applied across various domains. We hope to provide a concrete solution for the majority of fields, and therefore, the validation of synthetic datasets is crucial.

Q2: Question: Could the authors provide a breakdown of the types of errors observed in more detail? For example, what percentage of errors were due to hallucinations, misinterpretations, or other issues?

We have already provided the types of errors and their specific proportions in the experiments, along with some examples in the appendix(Line 506). The errors mentioned in the paper occur only in rare cases. We have collected data on the errors from the above-mentioned experiments. Despite using modules like Anomaly Management and Autonomous Feedback System to reject unreasonable content generated by the large model, and incorporating the Expert Analysis module for fine-tuning to reduce the model’s hallucinations, it is not possible to completely eliminate these issues. The occurrence of such errors is primarily due to the inherent limitations of the large model and the errors introduced by randomness. In future work, we will provide a more detailed discussion of the specific types of errors.

Q3: The paper lacks a discussion on the scalability of TestAgent, particularly when deployed at scale in real-world applications with a large number of test-takers.

Our paper focuses on proposing a universal testing framework that can transform existing tests into adaptive and intelligent testing formats. Regarding scalability, TestAgent has already transformed traditional rigid tests into role-playing agent-based tests. One obvious scalability feature is the use of prompt engineering, where different prompts are tailored to individuals of different ages and roles, ensuring the best testing outcomes. Moreover, existing question-generation plugins and other adaptive question-selection algorithms can also be integrated into TestAgent. The reason we have not provided a more extensive discussion on scalability is that it would make the discussion quite complex. Each component in Figure 2 can be extended as needed, so we only provide the basic framework, with modifications to be made on top of the corresponding components in Figure 2.

Q4: Question: How effectively does the Autonomous Feedback Mechanism handle ambiguous responses in practice, and are there thresholds for when a response is deemed too ambiguous?

For label determination, the paper provides a detailed explanation (Line494). For cases where label determination is difficult, we offer three rules to assist the large model in label assignment. These rules are based on three perspectives: domain relevance (Line187), response alignment (Line192), and logical coherence (Line197), to avoid such situations. Once the Autonomous Feedback Mechanism determines a failure, the corresponding measurement step is re-executed.

Our paper primarily focuses on proposing a novel testing framework that transforms existing tests into adaptive and intelligent testing formats. It aims to break through the bottlenecks of traditional testing and solve existing problems. This is the core of our paper. We appreciate the reviewer's suggestions, but some of the points raised are not relevant to this paper. For instance, the suggestions regarding large model performance , user experience analysis, and high-level fairness issues fall outside the scope of our work, as these are not directly related to the focus of our study. The suggestions you mentioned belong to other independent domains and are not related to our research. If you have any further questions, feel free to discuss them with us.

Thank you for taking the time to read this paper and provide your valuable suggestions. Your feedback is greatly appreciated. Below, we will address the concerns you raised.

Q1： The Universal Data Infrastructure is not well defined, I came away not really understanding this component. Please revise the description with a focus on clarity. Specifically cognitive diagnosis training.

The Universal Data Infrastructure consists of three main steps. First, Domain Verification is used to identify the testing domain. For example, for a mathematics ability test, the testing dimension is determined.

Next, in the Data Integration module, GPT is used to simulate students and generate response data. The data pair consists of Student - Question - Label. These response records are then passed on to the Cognitive Diagnosis Training module.

The purpose of Cognitive Diagnosis is to analyze the current student ability $\theta$ based on the response data. Taking the 3PL-IRT as an example, the model is expressed as: $P_i(\theta) = c_i + (1 - c_i) \cdot \frac{1}{1 + e^{-(a_i(\theta - b_i))}}$

Each question $i$ has characteristics such as discrimination $a$, difficulty $b$, and guessing parameter $c$. The goal of our Universal Data Infrastructure is to abstract the features $a$, $b$, and $c$ for each question across various types of tests. This allows the cognitive diagnosis model to analyze the data effectively.

Thus, Cognitive Diagnosis Training is the process of training the features $a$, $b$, and $c$. The loss function is as follows: $L(\theta,Y) = \sum_{i} -\left(y_i \log( P_i(\theta)) + (1 - y_i) \log(1 - P_i(\theta)) \right)$

This loss function calculates the cross-entropy loss between the true labels $y$ and the cognitive diagnosis $P(\theta)$, which is then used for training. This is the process of the Universal Data Infrastructure.

Q2: The major weakness here is that using an LLM introduces a new problem, namely LLM hallucinations leading to misrepresentation of results. The authors bring up this up in section 3.8. It would be worthwhile for the authors to discuss the trade off between the introduction of hallucinations and the issues of traditional testing that originally spurred the work.

Traditional testing is relatively rigid. Based on the capabilities of large models, we have developed the TestAgent framework to introduce innovation in testing formats, breaking free from the limitations of traditional tests. However, the issue of hallucinations in large models is unavoidable. We have tried to mitigate this effect by using modules such as Anomaly Management and Autonomous Feedback System, which help reject unreasonable content generated by the large model. Additionally, we incorporated an Expert Analysis module for fine-tuning to reduce the hallucinations. However, the hallucination issue cannot be completely eliminated. In future work, we plan to integrate expert evaluations into the framework to balance the hallucination problem, especially for high-stakes exams. The innovation of this paper lies in proposing an innovative testing framework that addresses issues like randomness and rigidity in traditional tests. We deeply appreciate your suggestions, which will be discussed in more detail in future work.