Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing

Thank you for your positive feedback and insightful suggestions, which is really important to us.

---

Method

Q1: What if evaluate GETA across more metrics?

In addition to concurrent validity, we verified GETA’s performance with three more metrics:

1. Stability - In App. D.2, we have analyzed GETA's performance stability against three factors: 1) item generator backbone, 2) the difficulty and 3) the number of seed items, for GETA’s initialization. The results in Tables 12 & 13 show that GETA maintains high validity across different hyperparameters and backbones, demonstrating its robustness and effectiveness. The convergence analysis in App. D.4, Fig. 6 also indicates that GETA converges faster and more stably than CAT, benefiting from selective generation.

2. Construct validity - We have examined it using the multitrait-multimethod (MTMM) matrix [1]:

   Method	GETA/Bias	i.i.d./Bias	GETA/Ethics	i.i.d./Ethics	GETA/Toxicity	i.i.d./Toxicity
   GETA/Bias	1
   i.i.d./Bias	0.9336	1
   GETA/Ethics	-0.0826	-0.2465	1
   i.i.d./Ethics	0.2051	0.1634	0.8283	1
   GETA/Toxicity	0.4007	0.5699	-0.2136	0.272	1
   i.i.d./Toxicity	0.6994	0.7818	-0.0616	0.427	0.8995	1

   where the heteromethod-monotrait and heteromethod-heterotrait Pearson correlations are in bold and Italic, respectively. It is expected that |heteromethod-monotrait corr| > |monomethod-heterotrait corr| > |heteromethod-heterotrait corr|, and the construct validity of this work is strong, as the averages are 0.8871 > 0.3449 > 0.3016.

3. Predictive validity - This metric evaluates a test based on its predictions for future, or technically, downstream/external tasks. In this sense, Va-O functions as predictive validity because: 1) The three OOD datasets, chosen as the reference measurement, were published approximately one year after the static data source; and 2) The tasks in the OOD datasets are more complex than those in the source datasets, which is elaborated on in App. B.1.2, L1424-1474. As shown in Fig. 3, GETA performs saticfactorily in predictive validity, particularly in social bias.

---

Experiment

Q2: Does the model-inferred difficulty accurately reflect the item difficulty?

1. Yes. To assess the generalization ability after pre-training, we conducted an experiment on the Pearson correlation between 25 evenly sampled difficulty levels and the corresponding actual difficulty, measured by AEP and EP (defined in App. B.1.1, L1230-1240). The item generator achieves correlations greater than 0.9 with GPT-3.5-Turbo and Gemini-1.0-Pro as examinee LLMs, demonstrating reliable generalization.
2. The definition of difficulty in this work is clarified in App. C.1, L1714-1726. Items answered incorrectly by most examinee LLMs are deemed highly difficult. Therefore, the items challenging for LLMs may not appear truly difficult for humans.

Q3: Is GETA effectively trained? Can GETA extrapolate and generate increasingly difficult test items?

1. Since the item generator may not initially map item parameters to items with high difficulty, during the test process, each newly generated item is answered by all examinees and re-calibrated by the VIRT model to compute its true parameters $\hat d$. Consequently, only items matching the specified parameters $d^*$ are used for ability estimation.
2. In the re-calibration process above, the item generator can discover increasingly difficult items due to the randomness of LLMs. These items are then collected to further fine-tune the generator, a process referred to as evolving in this paper.
3. Please refer to our response to Q2 for details on GETA's extrapolation ability, as half of the sampled difficulty levels exceed the coverage of the static data (not presented or more difficult).

Q4: Does GETA rely on a large-scale dataset for a specific value dimension?

As clarified in Sec. 4.1, L226, we use 5k calibrated items per value type with ~80 responses per item to train GETA. In other words, GETA requires modest training data and is still more data-efficient than other adaptive baselines.

Q5: Is GETA applicable to other criteria?

1. Yes. As discussed in App. A.1, L1087-1092, GETA is theoretically applicable to any well-defined and quantifiable criterion, provided that: 1) a sufficient number (~5k) of accessible test items for calibration and training, and 2) reliable evaluators to define the ground truth, allowing verification of the correctness of the responses.
2. The reason we focus on value conformity, as well as the scope and limitations of GETA, are initially discussed in the Impact Statement in the main paper. Due to space limits, further details are provided in App. A.1 and App. E.

---

[1] Campbell & Fiske. Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix. Psychological Bulletin, 1959.

Thank you for your thoughtful reviews and suggestions.

---

Claim

Q1: Why is variational IRT (VIRT) better than MLE-based IRT with fewer observations?

Theoretical

1. Variational Inference (VI) originated in statistical physics and was later generalized to probabilistic model research [1]. It aims to approximate an intractable posterior distribution using a simpler, tractable prior.
2. The feasibility of VIRT has been theoretically proven [2]. The key to VIRT's scalability is amortization [3], which reframes the per-observation optimization problem as a supervised regression task, sharing parameters across observations. VIRT estimates both item parameters and examinee abilities using a single forward pass of a neural network, imposing relatively low constraints on data size.
3. In contrast, MLE requires a sufficiently large number of observations to ensure consistency [4]. In typical IRT calibration, item parameters are estimated using MLE with extensive datasets of human responses, often involving thousands of participants [5][6]. However, in the context of LLM evaluation, the number of examinee LLMs is significantly smaller (eight LLMs and ~80 responses per item in GETA). As a result, MLE-based IRT would be considerably less stable in this setting.

Empirical

1. Empirical studies have consistently shown that VI is both fast and accurate in IRT, matching sophisticated methods like HMC in accuracy and simpler methods like EM in speed [2][7].
2. In Sec. 4.2, Table 2, the ablation study results show that GETA significantly outperforms its variant w/o VIRT, suggesting that VI enhances the stability and accuracy of IRT estimation with limited observations, thus effectively guiding the item generator and improving the overall performance.

---

Method

Q2: Is $a_i$ defined twice?

No. As defined in L140, $a_i$ generally refers to the examinee's ability. However, since GETA focuses on AI safety and ethics, $a_i$ is specifically narrowed down to LLMs' value conformity in L157.

---

Experiment

Q3: More data and details about the correlation coefficients?

1. The concurrent validity is represented by the Pearson correlation coefficient, linearly scaled to [0, 1]. Here, we further present the unscaled results of GETA, along with their confidence intervals:

   Value Type	Estimate	Va-L	Va-I	Va-O
   Bias	Pearson	0.8921	0.9336	0.6708
   	90% CI	[0.2627, 0.9889]	[0.7398, 0.9844]	[0.0765, 0.9134]
   Ethics	Pearson	0.8538	0.8346	0.6440
   	90% CI	[0.1065, 0.9847]	[0.4362, 0.9594]	[0.0294, 0.9053]
   Toxicity	Pearson	0.5849	0.8994	0.4366
   	90% CI	[-0.4568, 0.9501]	[0.6252, 0.9760]	[-0.2614, 0.8348]

2. Here is the concurrent validity measured by Kendall's Tau along with some unscaled VC scores used for correlation calculation: https://anonymous.sharefile.com/d-s2f613f41f8304cedad1a66d2a53418f3. Please feel free to request any additional data if needed.

Q4: Are the data samples used to calculate the Pearson correlation coefficients enough?

Thank you for raising this valuable concern!

1. The number of data samples used indeed influences the confidence interval of the correlation coefficient: generally, the more samples used, the lower the uncertainty in the estimated value.
2. Though the sample size of 8 is insufficient for most psychometric research, we believe it is reasonable in the context of LLM evaluation, particularly in this work, because:
   1. The correlations remain consistent across different experimental settings. Our sensitivity analysis in App. D.2, Table 12, shows that GETA's validity remains stable across various hyperparameters and generator backbones (12 settings in total), further supporting the reliability and robustness of the measure.
   2. Each data sample is aggregated from multiple observations and trials, minimizing noise. As stated in App. B.2.3, L1590, we collect $K=4$ responses per item for each examinee LLM in all adaptive tests. These responses are all used for ability estimation, and the estimated abilities are then aggregated. In static evaluation, we sample $K=10$ responses per item and aggregate the results to obtain the VC values as in App. B.1.1.

References

[1] Jordan et al. An Introduction to Variational Methods for Graphical Models. Machine learning, 1999.

[2] Wu et al. Modeling Item Response Theory with Stochastic Variational Inference. Arxiv 2021.

[3] Gershman et al. Amortized Inference in Probabilistic Reasoning. CogSci 2014.

[4] Newey & McFadden. Large Sample Estimation and Hypothesis Testing. 1986.

[5] Sharpnack et al. BanditCAT and AutoIRT: Machine Learning Approaches to Computerized Adaptive Testing and Item Calibration. Arxiv 2024.

[6] Ma et al. A novel computerized adaptive testing framework with decoupled learning selector. Complex & Intelligent Systems, 2023.

[7] Kim et al. Variational Temporal IRT: Fast, Accurate, and Explainable Inference of Dynamic Learner Proficiency. EDM 2023.

Thank you for your positive feedback and valuable insights, which is really important to us.

Due to space limits, we regret that we have to skip some questions and provide only partial responses before your reply.

---

Theory

Q8: Why simpler methods fail?

1. Static evaluation (SE) is the simplest approach. However, SE faces the chronoeffect challenge, which includes data leakage and difficulty saturation, as discussed in Sec. 1, L23-33. These issues can lead to the under- or overestimation of LLMs' ability.

2. We considered using advanced LLMs instead of the VIRT model to adjust item difficulty. However, we found it infeasible to generate items with specified difficulty without finetuning, which motivated us to incorporate psychometric methods into GETA.

   To demonstrate this, in social bias, we evenly sampled 25 difficulty levels and generated 20 items at each level using four generators: 1) the item generator of GETA, 2) the untuned backbone model of the item generator, LLaMA-3-8B, 3) GPT-4o, and 4) Gemini-1.5-Pro, with the latter three models being prompted with carefully crafted ten-shot examples. The 25x20=500 items were then presented to GPT-3.5-Turbo and Gemini-1.0-Pro, with each responding 10 times per item to assess their actual difficulty using AEP and EP (defined in App. B.1.1, L1230-1240). The Pearson correlation between the specified (intended) and measured (actual) difficulty is as follows:

   Examinee	Item Generator	w/ AEP	w/ EP
   	GETA's generator	0.9034	0.9385
   GPT-3.5-Turbo	LLaMA-3-8B	0.0935	0.1124
   	GPT-4o	-0.1712	-0.1501
   	Gemini-1.5-Pro	-0.3178	-0.4385
   	GETA's generator	0.9325	0.9041
   Gemini-1.0-Pro	LLaMA-3-8B	-0.1015	-0.0787
   	GPT-4o	-0.0172	0.1062
   	Gemini-1.5-Pro	-0.1234	-0.1170

   GETA demonstrates superior controllability over item difficulty compared to its backbone and two top-performing LLMs. This may be because: 1) Untuned LLMs struggle to grasp the relationship between item characteristics and difficulty parameters, resulting in random or even inverted difficulty levels; 2) Larger proprietary LLMs struggle to generate AI risk-related sensitive content, as this may conflict with established AI safety policies. Therefore, the sophisticated method of GETA are essential for addressing the chronoeffect challenge in moral evaluation.

3. We explain why MLE-based IRT fails in response to Q13.

Q9: What does $\hat p(y)$ in Eq. 4 mean?

1. As defined in Sec. 3.1, L146-151, $r$ is a textual response, while $y$ represents the (moral) correctness of $r$.
2. GETA is designed to probe the ethical boundaries of LLMs, so we want $\hat p(y)$ to represent the distribution of correct and incorrect responses when the LLM approaches its ability limit. This corresponds to the item difficulty equaling the examinee's ability in the context of CAT, so we convert $\hat p(y)$ to a restriction on the item parameter $d$ in $\mathcal{A}$ in Eq. 6.

Q10: Why use an indicator function with $\epsilon$ in Eq. 6?

The item generator may not accurately map the specified item parameters to items at first, so we expand $d^*$ to $\mathcal{A}$ as a fault-tolerant mechanism. During the test, each newly generated item is responded to by all examinees and re-calibrated by the VIRT model to compute its true parameters $\hat d$. The indicator function ensures that only items matching the specified parameters are used for ability updating.

---

Experiment

Q13: Why the MLE variant fails?

1. In the variant w/o VIRT, we use the MLE-based IRT estimator implemented in the CAT baseline instead of VIRT estimators. Following typical practice, we re-estimate all parameters, including item parameters and examinee abilities, using static data and new test records (new items and responses) at each iteration.
2. Please refer to Q1 by Reviewer nS3h for details on the instability of MLE with limited observed data. As a result, item parameters and abilities may fluctuate across iterations, misleading the item generator and leading to poor performance.

Q15: What are the confidence intervals for the Va?

Please refer to Q3 & Q4 by Reviewer nS3h.

Q17: Are the i.i.d. items likely to favor GETA?

Thanks for your valuable insight! It makes sense.

1. All measurements in this paper share the same data source introduced in App. A.2. In this sense, we consider the GETA-generated items, whether before or after paraphrasing, as i.i.d.
2. To measure the distribution shift from static data to these i.i.d. items, we compute the semantic distance between the two item sets: JS Divergence = 0.0129, average cosine distance = 0.0791, and maximum mean discrepancy = 0.0116 (bandwidth = 2), indicating negligible bias in this work.

---

Please feel free to ask further questions or request more detailed responses if there's anything we can do to address your concerns.

Thank you very much for your insightful comments.

We sincerely appreciate your efforts in reviewing our paper. As GETA integrates psychometric methods, we have included most of the details in the appendix due to space limits.

---

Method

Q1: More details of GETA (training, inference, and data)?

1. The details about the training of the VIRT estimators and item generator are presented in App. B.2.
2. A comprehensive running example of Mistral-Medium in the bias test is provided at the end of App. C.4 (L1922-1956), with an illustration of the key variables and components on the left of Fig. 8.
3. Regarding the methodology, the entire process of traditional CAT is introduced in App. C.1 as a theoretical basis. Sec. 3.2 and 3.3 (L205-274) are further elaborated on theoretically in App. C.2 and C.3, respectively.
4. All baselines and variants share the same static data source as GETA, detailed in App. A.2.

Q2: How are new items generated?

As described in App. B.2.2, L1570-1573, the item generator $p_\omega(x|d)$ takes a 2-d vector $d=(b,c)$ as input and generates the item $x$ in an auto-regressive manner. Here are two examples of the generator's inputs and outputs in the context of toxicity:

Q3: Can the discussion of ELBO in the main paper be reduced?

1. ELBO serves as the objective in variational IRT estimation, significantly improving GETA's performance and making it a crucial component of our methodology.
2. A more complete derivation of ELBO is provided in App. C.2, L1758-1825. In this work, we approximate a joint distribution $p(x,y)$ and apply Jensen's inequality twice, and the version in the main paper has been significantly simplified.

---

Experiment

Q4: Are Va-I and Va-O fair?

We believe the comparisons of Va-I and Va-O are fair.

1. For Va-I and Va-O, we use i.i.d. items and OOD datasets, respectively, as reference measurements, both of which are static data. We use three implementations of VC over static data — EMT, AEP, and EP — detailed in App. B.1.1, all of which are widely adopted in AI safety and ethics research [1][2][3].
2. In the main paper, we report EP-based VC, as it sets the highest safety standard: given an item, an LLM is considered safe only if it generates no responses violating human values in all $K$ trials.
3. We also report Va-I and Va-O results with different VC estimation methods in App. D.1, Table 10. These results remain stable across different VC estimation methods, supporting the fairness and reliability of the comparisons.

Q5: Why focus on ethics evaluation?

We have discussed the significance of advancing ethics evaluation in the Impact Statement (L442-456) and the evaluation scope of GETA in App. A.1.

---

Others

Q6: More analysis highlighting GETA's superiority?

In addition to our main results, we conduct a series of analysis experiments in Sec. 4.3 and App. D.2-D.4:

1. Item novelty analysis (Fig. 4) - GETA generates novel items comparable to human-crafted ones in diversity and quality, effectively addressing data leakage.
2. Difficulty adaptivity analysis (Fig. 4) - GETA better differentiates LLMs of different families and versions with adjustable difficulty, alleviating data saturation.
3. Stability analysis (Table 12 & 13) - GETA consistently outperforms most baselines across various hyperparameters and generator backbones, suggesting its stability and robustness.
4. Human study (Table 14) - GETA correlates better with human judgment.
5. Efficiency analysis (Fig. 6) - GETA converges faster and more stably, showing greater efficiency.

Q7: If static datasets easily saturate, Why can GETA generate more difficult items?

1. Static datasets easily saturate because they contain few challenging items. For example, only 1.2% of the data in RealToxicityPrompt [1] were labeled as challenging for most LLMs, not to mention that it was published in 2020.
2. GETA's generator learns the mapping from difficulty levels to items, allowing it to generate a large volume of items at higher difficulty levels, thus elevating the overall difficulty of the test.
3. For the difficulty of individual items, please refer to Q3 by Reviewer HT7s for an explanation of why GETA can discover increasingly difficult items.

---

References

[1] Gehman et al., RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. EMNLP 2020.

[2] Wang et al., ToViLaG: Your Visual-Language Generative Model is Also An Evildoer. EMNLP 2023.

[3] Pozzobon et al., On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research. EMNLP 2023.