Reliable and Efficient Amortized Model-based Evaluation

Dear reviewer 9Mrp,

Thank you for your valuable feedback. First, we would like to address the weakness below.

Weaknesses 1: While using Item Response Theory (IRT) to model the evaluation process is valuable, it also introduces an additional understanding barrier for those unfamiliar with these concepts. For example, the term "item parameters" could be simplified to "difficulty," giving readers a more intuitive sense of what the modeling aims to capture.

Answer 1: We sincerely thank the reviewers for their insightful suggestions. To ensure clarity and accessibility for readers, we have incorporated the following clarification into the introduction part, where the terms “item” and “item parameter” first appear:

The terms "item" and "question" are used interchangeably. In this paper, we exclusively use Rasch's model~\citep{rasch1993probabilistic}, a fundamental and straightforward approach within IRT, where the "item parameter" represents the difficulty level of a question.

We greatly appreciate this feedback, as it has helped us enhance the paper's readability.

Weaknesses 2: One of the main claims made in the paper is the emphasis on "long-term, adaptive monitoring as models evolve, going beyond static benchmarking" (Line 157). However, the paper does not provide sufficient experimental evidence to support this claim. For example, it does not introduce new models to demonstrate adaptive monitoring and instead relies on conventional approaches like train/test splits. Additionally, the conditional item generator is only tested on a single dataset, which limits the generalizability and persuasiveness of the results.

Answer 2: We appreciate the insightful feedback. In our work, IRT inherently supports model monitoring by enabling the evaluation of new models. Specifically, evaluating new models can turn into monitoring when these new models represent different versions of the same model over time. Actually, experimental evidence for monitoring does exist in our results. For instance, we evaluated multiple versions of OpenAI's GPT-3.5 (0125, 0301, 0613, and 1106) on the AIR-Bench dataset. The results demonstrate that the IRT ability parameter on AIR-Bench changes over time: -0.63 (Jan 25, 2023), 0.79 (Mar 1, 2023), 0.99 (Jun 13, 2023), and 0.02 (Nov 6, 2023). These findings suggest that GPT-3.5 became progressively safer from January to June but exhibited a notable decline in safety performance with the November update. We hope this illustrates how IRT can reliably and efficiently monitor model performance as it evolves.

Additionally, we acknowledge that the conditional item generator has been tested on a single dataset in the current version of our work, which limits the scope of generalizability. To address this, we have extended our approach by training a generator that can produce questions based on specified difficulty levels and desired datasets. Specifically, we combined the response matrices of 24 datasets into a single, big response matrix and conducted the joint calibration to develop a regression model that predicts item difficulty from the question embeddings. Note that each question in one single dataset is prefixed with a brief tag introducing the dataset itself to provide context information. Using this regression model, we trained the item generator. Preliminary results from this approach are promising, and we are working diligently to provide detailed experimental results before the rebuttal deadline.

We sincerely appreciate your thoughtful feedback, which has greatly helped us refine and strengthen our work.

Weaknesses 3: Another critical issue is the practicality of the proposed system. Traditionally, NLP benchmarks included large datasets, making them easier to encode for analysis. However, recent evaluation trends are shifting toward more challenging tasks with relatively smaller datasets (e.g., using AIME data for testing). This work does not convincingly demonstrate its relevance or necessity in light of this trend.

Answer 3: We respectfully acknowledge the concern regarding the relevance of our approach given the trend toward smaller, more challenging datasets in recent evaluation tasks. However, to the best of our knowledge, current evaluation frameworks, including widely adopted standards, continue to rely extensively on large datasets for comprehensive analysis. Furthermore, while some very recent specialized benchmarks are small, the vast majority of benchmarks (both old and new) emphasize large sizes, e.g., for coverage. For instance, AIR-Bench, a recent safety benchmark, includes 5,694 questions, requiring significant computational resources to ensure thorough assessment. We believe this underscores the ongoing importance of approaches that can effectively handle large-scale evaluations.

Dear reviewer V1AP,

Thank you for your valuable feedback. We would like to address the weakness below.

Weaknesses 1: Some limitation in generated questions quality not fully addressed. Need more discussion about potential risks of automated question generation such as bias towards model-generated questions

Answer 1: We sincerely appreciate the reviewer’s thoughtful feedback on the quality of generated questions and the potential risks associated with automated question generation. We understand the importance of validating the quality of generated questions, especially the potential risk such as bias or harmful contents. To provide clarity, we have made samples of the generated questions available for review at Generated Questions. These examples demonstrate that the format, style, and content of the generated questions align closely with those in the original datasets. Looking ahead, to automate the validation process in the absence of human oversight, we plan to use advanced models such as GPT-4 to verify semantic correctness before incorporating the questions into evaluation workflows. However, human reviewers remain indispensable in identifying, refining, and mitigating any biases that may arise in AI-generated content. This collaboration allows us to leverage the complementary strengths of both parties.

Regarding the potential risks of automated question generation, we would like to first underscore the primary motivation for incorporating an automated question generator. It plays a supplemental role in adaptive testing, where questions are selected iteratively based on a desired difficulty level to evaluate model performance. While the primary source of questions is the original item pool, there may be instances where questions at a specific difficulty level are exhausted. In such cases, the item generator is employed to create additional questions. Importantly, this generator constitutes only a minor component of the evaluation process, with the majority of questions still derived from the original pool, thereby limiting the potential bias introduced by model-generated questions.

We also acknowledge that automated question generation could serve broader purposes, such as replacing overused questions, expanding datasets, or constructing entirely new datasets. In these contexts, the potential for bias exists. However, as evidenced by modern AI benchmarks such as AIR-Bench [1], HEx-PHI [2], AdvBench [3], and SaladBench [4], the use of AI in dataset development is becoming increasingly common. A widely adopted practice is to involve human reviewers in the process to mitigate potential biases. We hope this addresses the reviewer’s concerns, and we are happy to further clarify or elaborate if needed.

[1] Zeng, Y., Yang, Y., Zhou, A., Tan, J. Z., Tu, Y., Mai, Y., ... & Li, B. (2024). Air-bench 2024: A safety benchmark based on risk categories from regulations and policies. arXiv preprint arXiv:2407.17436.

[2] Qi, X., Zeng, Y., Xie, T., Chen, P. Y., Jia, R., Mittal, P., & Henderson, P. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. In The Twelfth International Conference on Learning Representations.

[3] Zou, A., Wang, Z., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

[4] Li, L., Dong, B., Wang, R., Hu, et al. (2024). Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. ACL 2024.

Weaknesses 4: Need more baseline comparisons with other evaluation methods (like Elo scores of LLM arena)

Answer 4: Thank you for raising this insightful concern. We explored the possibility of incorporating Elo scores from the LLM arena into our comparisons but found that Elo scores in the LLM arena are not available for individual datasets. However, we would like to emphasize that our work includes extensive evaluations of models across 24 datasets using IRT, with comparisons to other evaluation methods including Classical Test Theory (CTT) scores and HELM leaderboard scores. For a specific dataset, the CTT score for a model is calculated as the average score across all its responses, while the HELM scores are crawled from the HELM leaderboard.

In Appendix A, Figure 8 (second and third) illustrate the correlation between the IRT ability parameter $\theta$ obtained through traditional calibration and both the CTT and HELM scores. As shown, there is generally a strong correlation between $\theta$ and CTT/HELM scores across most datasets. Our findings underscore the robustness and reliability of the IRT-based evaluation framework, highlighting its alignment with established methodologies and its ability to provide valuable insights into model performance. We hope this clarification adequately addresses your concern regarding the breadth of our baseline comparisons.

Weaknesses 5: While the paper claims that it is model-based evolution and decoupled model performance from the test subset selection, it still needs to test models on some examples, which is a major concern of mine

Answer 5: Thank you for raising this important concern. We would like to clarify our approach. Evaluating the entire dataset across all models is often prohibitively expensive, which is why major leaderboards, such as HELM, typically evaluate only a subset of the dataset and report an average score. However, this practice introduces variability, as different leaderboards may sample different subsets, potentially leading to fluctuations in model rankings depending on the difficulty of the subsets. To address this issue, we employ an IRT-based method that allows for consistent estimation of model performance across different subsets. This approach retains the efficiency benefits of subset testing while ensuring the “item-invariance of examinee’s ability estimate,” a property that traditional methods, such as reporting an average score, do not offer. By decoupling model performance from the specific test subset, we are not suggesting that our approach does not make use of any test subset, but rather that the ability estimate remains invariant across different subsets of questions. For a more detailed explanation of this property, we refer readers to standard psychometric texts, such as Baker (2001), Chapter 5, page 90. We hope this clarifies our methodology and demonstrates the robustness of our approach. Thank you again for your thoughtful feedback, and please let us know if you have any further questions.

[Baker 2001] Frank B Baker. The basics of item response theory. ERIC, 2001. Available online at https://eric.ed.gov/?id=ED458219

Thanks for the detailed response from the authors, I have raised my scores.

Dear reviewer gy8L,

Thank you for your valuable feedback. First, we would like to address the weakness below.

Weaknesses 1: Some concerns about experiment setting and base model selection.

Answer 1: We will further address this in the Question part.

Weaknesses 2: This work mainly develops techniques to improve IRT-based evaluation. However, it doesn't use it to run systematic evaluations of current models, so readers can not cross-validate with other evaluation methodologies or their own experience.

Answer 2: Thank you for your thoughtful feedback. We would like to clarify that our work includes extensive evaluations of models across 24 datasets using IRT, with comparisons to established evaluation methodologies including Classical Test Theory (CTT) scores and HELM leaderboard scores. For a specific dataset, the CTT score for a model is calculated as the average score across all its responses, while the HELM scores are crawled from the HELM leaderboard.

In Appendix A, Figures 8 (second and third) illustrate the correlation between the IRT ability parameter $\theta$ obtained through traditional calibration and both the CTT and HELM scores. As shown, there is generally a strong correlation between $\theta$ and CTT/HELM scores across most datasets. Additionally, Appendix A, Figure 10 (second and third), provides an analysis of the correlation between $\theta$ and CTT/HELM scores when using the joint calibration approach. We find that results from amortized calibration are comparable to those obtained with traditional calibration methods.

We hope this explanation addresses the concern regarding the systematic evaluation of models. Our results demonstrate the robustness of the IRT-based evaluation framework and its alignment with established methodologies, providing valuable insights into model performance. Thank you again for highlighting this aspect, and we appreciate the opportunity to clarify.

Regarding your questions, we address them below:

Question 3: How do you validate the quality and correctness of the generated questions? Specifically, how do you deal with questions that might match the desired difficulty but be semantically invalid, without human oversight?

Answer 3: Thank you for your insightful question. We understand the importance of validating the quality and correctness of generated questions, especially in cases where they may match the desired difficulty but lack semantic validity. To provide clarity, we have made samples of the generated questions available for review at Generated Questions. These examples demonstrate that the format, style, and content of the generated questions align closely with those in the original datasets. Additionally, we conducted human evaluations on all generated questions, confirming that they are semantically valid and align well with their intended purpose. Looking ahead, to automate the validation process in the absence of human oversight, we plan to use advanced models such as GPT-4 to verify semantic correctness before incorporating the questions into evaluation workflows. From a theoretical perspective, the embedding model used in our approach (currently Llama 3 8B) plays a crucial role in distinguishing semantically valid questions. Since semantically invalid questions would likely produce embedding vectors that deviate significantly from those of valid ones, their predicted difficulty would also diverge from the target level. Consequently, such invalid questions are less likely to be selected during the generation process, as they would be outperformed by semantically valid questions with more accurate difficulty predictions. We sincerely appreciate your thoughtful feedback, which helps us enhance the robustness and clarity of our methodology. Thank you for engaging with our work so constructively.

Question 4: Did you compare the amount of compute needed to create the amortized model with the compute needed to evaluate a single model? How many models do you need to evaluate to have the amortized model be more efficient?

Answer 4: Thank you for your excellent question. Our proposed framework consists of two phases: calibration and scoring. The creation of the amortized model is done in the calibration phase while evaluating a single LLM is in the second phase.

Firstly, in the calibration phase, we build the amortized model using responses from dozens of LLMs, which are sourced from existing leaderboards and benchmarks. To create this model, we first compute embeddings for all available questions across our datasets (each question corresponds to a 4096-dim vector) and then optimize the parameters of the amortized model using techniques such as EM or MLE. The embedding model we use is Llama 3 8B, and the amortized model is a simple 2-layer 4096-hidden fully-connected network. This step incurs a one-time cost, which is much smaller than evaluating a single LLM of comparable size, as the cost of inferring embeddings is equivalent to generating one token using an LLM, and the cost for training the amortized model is neglectable in comparison to the LLM inference cost.

Secondly, in the scoring phase, while evaluating new LLM, we need the question difficulties to infer ability. The challenge arises when the questions are new such as when we need to replenish the item bank with new questions due to contamination or the question overused. Then, if we have an amortized network obtained from phase one, the computational cost for inferring difficulty is number_of_questions x (embedding_per_question + forward pass cost for 2-layer network). In contrast, without the amortized model, estimating question difficulty would typically require running dozens of LLM models (ranging in size from small 3B to larger 70B) on those new questions to gather sufficient data for reliable estimation. We would also like to emphasize that running an LLM on a question involves multiple forward passes (equal to the number of generated tokens), and evaluating the outputs of the LLM incurs additional costs, such as querying GPT-as-a-Judge or human annotators, which is very expensive. If we are not concerned about evaluating the LLMs on new questions, we already have the question difficulty (obtained from the calibration phrase). Thus, we do not need the amortized network.

Therefore, the amortized model demonstrates significant computational savings, particularly when evaluating newly added questions. In summary, while training the amortized model incurs a small upfront computational cost, this investment pays off substantially in terms of efficiency for subsequent evaluations. We hope this explanation provides clarity and effectively addresses your concerns. Thank you again for your thoughtful question—it has allowed us to highlight the practical benefits of our approach.

Dear reviewer c8mZ,

Thank you for your valuable feedback. First, we would like to address the weakness below.

Weaknesses 1: The most glaring weakness of the paper is that they claim to test 180 models and don’t show the results of all of them anywhere in the paper or an appendix. The closest I could see of all of the models evaluated was in Figure 1, but it is truncated to fit in the paper

Answer 1: Thank you for your detailed feedback and for highlighting this critical point. To address your concern, we have provided a comprehensive table listing all models evaluated in our study. The full table is accessible at the following links: model list part 1, model list part 2, and model list part 3.

To further enhance transparency, we have also included a dataset-model matrix to document the presence of models across different datasets. In this matrix, rows represent datasets, columns represent models, and blue blocks indicate that a specific model is evaluated on a given dataset. dataset-model presence matrix

We want to point out that the evaluation results of these models are available on HELM. The value of our work is making LLM evaluation more efficient and reliable through the use of IRT. In this paper, we have demonstrated that the result from our method is highly agreeable with HELM as indicated in Figure 4. We appreciate your thoughtful review, which has helped us improve the clarity and transparency of our work. Thank you for providing the opportunity to address these important concerns.

Weaknesses 2: a table or chart at least demonstrating the model performance on the conditionally generated dataset and comparing it to the subsets of HELM would have been useful.

Answer 2: At first, we appreciate your insightful question. We are currently conducting experiments to address this and are evaluating the performance of several models on the newly generated AIR-Bench-style questions. These results will be compared with their performance on the original AIR-Bench questions to provide a clearer understanding of the differences and insights gained.

Weaknesses 3: Some formatting concerns: the references aren’t linked the figures aren’t linked some of the graphs are hard to read when printed due to small text

Answer 3: We thank you for highlighting these formatting issues. We have carefully addressed all the concerns in the revised submission. Specifically, we have ensured that references are properly linked, figures are correctly linked, and graphs are adjusted to enhance readability. We greatly appreciate your feedback, which has helped us improve the clarity and presentation of our work.

Dear reviewer D8pS,

Thank you for your valuable feedback. We would like to address the questions below.

Question 1: Generating questions for adaptive testing is clearly useful, but what do the the generated questions look like?

Answer 1: Thank you for your thoughtful and insightful question. To provide further clarity, we provide generated questions from our finetuned generator(which is trained on all questions from all datasets). The generated questions are accessible at Generated Questions. These examples illustrate that the format, style, and content of the generated questions align well with those of the questions in the original datasets. We sincerely hope this addresses your concern, and we deeply appreciate your interest in our work.

Question 2: How does the underlying generative model affect the results? Thank you for your thoughtful question. To address it, we conducted an ablation study to investigate the performance of item generators based on different underlying generative models. Here is the design of our experiment: we fine-tuned two item generators using the same fine-tuning procedure but with different base models. One generator utilized Llama3 8B, while the other used Mistral 7B v0.3. Each generator produced a distinct AIR-Bench-style question pool containing 1,000 questions. Together with the original AIR-Bench questions, these pools were used to query 35 language models, comprising 27 training models and 8 testing models.

The models' responses were then converted into binary patterns using GPT4o-as-a-judge, which assigned a score of 0 (failure) when the model completed an unsafe request and a score of 1 (success) when the model declined the request. This process generated three response matrices—one for each question pool (original AIR-Bench, Llama3-generated questions, and Mistral-generated questions), denoted as $Y_1$, $Y_2$, and $Y_3$, respectively. These matrices were concatenated along the question dimension to create $Y_{all}$. For $Y_1$, the section corresponding to the 27 training models is denoted as $Y_{1, train}$, and the section for the 8 testing models as $Y_{1, test}$. Similarly, we define $Y_{2, train}$, $Y_{2, test}$, $Y_{3, train}$, $Y_{3, test}$, $Y_{all, train}$, and $Y_{all, test}$. Subsequently, we performed calibration on $Y_{all, train}$ to estimate the difficulty parameters of the questions, $z_{all, train} = [z_{all, train, 1}, z_{all, train, 2}, z_{all, train, 3}]$, and the ability parameters of the models, $\theta_{all, train}$. This joint calibration was necessary because question difficulty is a relative attribute, making meaningful comparisons possible only when all sets are calibrated together. For reference, calibration was also performed solely on the original AIR-Bench responses ($Y_{1, train}$), producing difficulty estimates $z_{org, train} = [z_{org, train, 1}]$ and ability estimates $\theta_{org, train}$. For the 8 testing models, we inferred their ability parameters using the difficulty parameters and responses from the original AIR-Bench ($z_{org, train, 1}$ and $Y_{1, test}$), denoted as $\theta_{org, test, 1}$. Similarly, we inferred the ability of the testing models using $z_{all, train, 2}$ and $Y_{2, test}$, as well as $z_{all, train, 3}$ and $Y_{3, test}$, resulting in $\theta_{all, test, 2}$ and $\theta_{all, test, 3}$, respectively. The correlations between $\theta_{org, test, 1}$ and $\theta_{all, test, 2}$, as well as between $\theta_{org, test, 1}$ and $\theta_{all, test, 3}$, were both 0.81. These results demonstrate that the choice of base model for the item generator does not significantly impact the ability estimates of the testing models. We hope this study addresses your question thoroughly. Thank you for your insightful feedback, which has contributed to the robustness of our work.

Question 3: How can we automatically verify the generated questions are valid/correct?

Answer 3: Thank you for raising this important question. Ensuring the correctness of generated questions involves multiple dimensions, including semantic accuracy and alignment with the target difficulty. For example, in a safety dataset like AIR-Bench, it is essential that the generated questions effectively measure safety. To assess semantic correctness, we conducted human evaluations on a subset of the generated questions, confirming their validity. Moving forward, we aim to automate this validation process by leveraging advanced models such as GPT-4 to verify semantic correctness before integrating the questions into evaluation workflows. We appreciate your thoughtful feedback, which has helped us refine and clarify our methodology. Thank you again for your question.

Weaknesses 4: The experiments predominantly rely on synthetic setups or controlled environments. Real-world case studies or collaborations with industry practitioners would strengthen the empirical validation.

Answer 4: Thank you for raising this concern regarding our experimental setup. Our experiments were conducted on 24 real-world datasets sourced from HELM and evaluated across various state-of-the-art LLMs, including GPT-4 and Llama 3.1. For instance, both the calibration analysis and the test-independent ability estimation were performed using these real-world datasets. The only semi-synthetic component in our study pertains to the computerized adaptive testing (CAT) experiment. This approach is commonly used in active learning research, as recruiting new test takers for every experiment is often impractical. We greatly value your feedback and would be delighted to hear any suggestions for additional real-world experiments that could further strengthen our analysis. Your input would be invaluable in ensuring our work has the greatest possible impact and relevance.

Regarding industry collaborations, we see significant potential for impactful partnerships with HELM, a leading large-scale leaderboard for AI model evaluation. Currently, HELM utilizes Classical Test Theory (CTT) scores on subsets to manage evaluation costs. However, our findings suggest that this approach may lack reliability in certain scenarios. Our proposed IRT-based method offers a more reliable alternative while preserving the efficiency of subset testing, making it a practical and scalable solution for large-scale evaluation efforts. By integrating IRT into frameworks like HELM, we aim to enhance the robustness and accuracy of model evaluations, ultimately benefiting the broader AI community. We sincerely appreciate the opportunity to address these points and to emphasize the practical applicability and impact of our research. Thank you for your thoughtful consideration and support!

Dear Reviewer tNxg,

Thank you for your valuable feedback. We answer your comment below.

Weaknesses 1: The introduction of IRT and amortized calibration adds complexity to the evaluation process, which might be challenging for practitioners without a background in psychometrics.

Weaknesses 3: The integration of Item Response Theory (IRT) with machine learning evaluations assumes a level of prior knowledge (e.g., psychometric concepts) that could limit accessibility.

Answer 1 & 3: Thank you for your thoughtful feedback and for highlighting the potential challenges of integrating Item Response Theory (IRT) and amortized calibration into machine learning evaluation frameworks, especially for practitioners less familiar with psychometric methods. We appreciate the opportunity to address these concerns and share our plans for making these methodologies more accessible and impactful.

To bridge this gap, we aim to integrate IRT into widely-used generative model evaluation frameworks such as HELM and LM-eval-hardness. For example, the computerized adaptive test application of IRT demonstrated in our paper could be implemented as a built-in function within a dataloader in HELM. This integration, along with comprehensive documentation, will be made publicly available to support ease of use and reproducibility. We welcome suggestions for other evaluation frameworks where this methodology could add value and would greatly appreciate any recommendations from the reviewer community.

Recognizing the importance of accessibility and reproducibility, we are prioritizing a user-friendly and well-documented implementation. To support further exploration and development, we have made our resources openly available on our GitHub repository: Anonymous Repository. This repository aims to serve as a foundation for extending evaluation methodologies across machine learning, NLP, and related domains.

Finally, while IRT may be new to some in the ML community, it is firmly rooted in latent variable modeling—a foundational topic covered in many standard machine learning textbooks. For example, Chapter 12.2.4 of Pattern Recognition and Machine Learning provides an excellent overview and is accessible here. We hope this context encourages its adoption and sparks further interest in its potential applications.

Thank you again for your questions and suggestions, which have been invaluable in refining our approach and thinking about broader community engagement.

Weaknesses 2: Details on the practical implementation of the item generator and its integration into existing workflows are limited.

Answer 2: Thank you for bringing up this important point. In response, we have revised Section 3 to provide additional details on the practical implementation of the item generator and its integration into existing workflows. Specifically, the item generator addresses gaps in the item bank by creating new items when no existing ones meet the desired difficulty level. For example, if a target difficulty level $z$ is specified and no item in the bank aligns closely with it, the generator produces a new question tailored to the specified difficulty. These newly generated questions are then validated—either through LM-as-a-judge mechanisms or by human evaluators—to ensure their quality and relevance. Once validated, the items are added to the bank for future use, seamlessly enhancing the evaluation workflow. We believe these clarifications highlight the practical utility of the item generator and its role in streamlining and improving evaluation processes. Thank you again for your thoughtful feedback—it has been instrumental in improving the clarity and completeness of our work.

Thanks for your responses, which address my previous concerns. I will update my score.

Regarding your questions, we address them below:

Question 1: The paper mentions missing values in the response matrix. How are these handled during calibration and scoring to ensure accurate ability estimation?

Answer 1: Thank you for your insightful question. During both calibration and scoring, missing values are masked out when computing the likelihood, ensuring that they do not influence the calculations. Additionally, to mitigate potential numerical issues in our experiments, we exclude questions from the response matrix where all models either answer correctly or incorrectly. This step helps maintain the reliability of ability estimation and ensures the robustness of the evaluation process. We hope this explanation addresses your question. Please let us know if further details would be helpful.

Question 2: Is the IRT model effective for non-text data, such as images or multimodal datasets? What adjustments are necessary?

Answer 2: Yes, the IRT model can be applied to non-text data, including images and multimodal data. Currently, the IRT model we are studying operates on binary responses, which makes it naturally suited for tasks where evaluation outcomes can be categorized as success (1) or failure (0). For example, for image data, we need a grading method to assess the generated images from the model, which is a tool that the community has already had (such as vision-language-model-as-judge or human evaluator). Although we study the setting of binary response, we want to point out that there are IRT models tailored to non-binary metrics or tasks requiring finer-grained assessments [1]. This flexibility allows the IRT framework to adapt to a wide range of datasets and evaluation scenarios, making it applicable to diverse modalities. Expanding the model to address these adjustments is a natural extension of our work, and we are eager to explore this direction in future research.

Question 3: How does the automated question generation avoid introducing bias, ensuring fair and objective evaluations?

Answer 3: Thank you for this insightful question. We would like to first underscore the primary motivation for incorporating an automated question generator. It plays a supplemental role in adaptive testing, where questions are selected iteratively based on a desired difficulty level to evaluate model performance. While the primary source of questions is the original item pool, there may be instances where questions at a specific difficulty level are exhausted. In such cases, the item generator is employed to create additional questions. Importantly, this generator constitutes only a minor component of the evaluation process, with the majority of questions still derived from the original pool, thereby limiting the potential bias introduced by model-generated questions.

We also acknowledge that automated question generation could serve broader purposes, such as replacing overused questions, expanding datasets, or constructing entirely new datasets. In these contexts, the potential for bias exists. However, as evidenced by modern AI benchmarks such as AIR-Bench [2], HEx-PHI [3], AdvBench [4], and SaladBench [5], the use of AI in dataset development is becoming increasingly common. A widely adopted practice is to involve human reviewers in the process to mitigate potential biases.

We will revise our paper to emphasize the necessity of a collaborative framework where human experts and the automated item generator work together to ensure fairness and objectivity. The item generator excels in leveraging embedding representations to create questions at specific difficulty levels, a task that may surpass human intuition in precision and consistency. However, human reviewers remain indispensable in identifying, refining, and mitigating any biases that may arise in AI-generated content. This collaboration allows us to leverage the complementary strengths of both parties.

We deeply appreciate the opportunity to address this important consideration and remain committed to advancing robust, fair, and efficient evaluation frameworks through continuous improvement and ethical practices. Thank you once again for highlighting this critical aspect of our work.

[1] Ostini, R., & Nering, M. L. (2006). Polytomous item response theory models (No. 144). Sage.

[2] Zeng, Y., et al. (2024). AIR-Bench 2024: A safety benchmark based on risk categories from regulations and policies. arXiv preprint arXiv:2407.17436.

[3] Qi, X., et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. In The Twelfth International Conference on Learning Representations.

[4] Zou, A., et al. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

[5] Li, L., et al. (2024). Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. ACL 2024.