A Closed-Form Solution for Fast and Reliable Adaptive Testing

We sincerely appreciate your positive feedback on the readability, theoretical clarity, and methodological contributions of our work.

Q1: if other CDMs are adopted, such as neural network-based CDMs or the DINA model, does the submodular property still hold? Can greedy algorithms still be applied?

A1: That's a great and insightful question. To answer directly: when using more complex models like neural network CDMs or the DINA model, the submodular property generally does not hold. However, greedy algorithms can still be applied effectively in practice.

First, we focus on IRT in the main text because it's the most widely used measurement models for adaptive testing, especially due to its simplicity and interpretability. It's the foundation for many large-scale tests like the GRE. This is why all the background, related work, and theoretical analysis in our paper are exclusively based on the IRT setting. Under IRT, the estimation objective has nice mathematical properties (e.g., strong convexity and bounded Hessian), which allow us to derive an approximate submodular structure. This is the basis for the theoretical guarantees in our Lemma and Theorem.

In contrast, models like neural networks are highly non-linear and non-convex. Their loss functions are generally not convex as you said, let alone strongly convex, so the theoretical results in our paper do not apply to these models. As a result, all the theoretical analysis in the paper is based solely on the IRT model.

However, greedy algorithms can still be used as heuristics even without submodularity guarantees. In fact, many non-convex problems in practice have been shown to exhibit behavior similar to submodular functions, at least empirically [1][2]. If we can define some notion of "informativeness" or "uncertainty", greedy selection can still be a powerful strategy, e.g., active learning, coreset. Moreover, Table 1 demonstrates that even without theoretical guarantees, greedy selection still performs well in practice when applied to neural network methods. We'll highlight this point in the next version.

Q2: The design (e.g., Hessian stability term) assumes smooth and twice-differentiable losses. This may not hold for discrete CDMs (e.g., DINA) or models with non-differentiable components.

A2: Yes, the majority of the design in our paper is grounded in IRT. This is because IRT remains the dominant framework in most large-scale testing, as discussed in motivation and related work section. As you noted, for discrete CDMs, there is currently a lack of reliable theoretical foundations in this domain. While our analysis centers on IRT, we do consider complex approaches as well. For example, to calculate Hessian in complex models like NeuralCDM, computing the exact Hessian is often infeasible due to the high dimensionality and nonlinearity of the parameter space. In these cases, we use a quasi-Newton approximation of $\mathcal{H}^{-1}$, as described in Appendix A.

Q3: The author(s) assumes fixed guessing/slipping probabilities $\pi_g$ and $\pi_s$, which may not confirm with real-world behaviors.

A3: Thank you for your comment. Yes. In real-world settings, guessing and slipping behaviors can vary across different examinees and items. In our work, we use fixed values for $\pi_g$ and $\pi_s$ as a simplifying assumption to make the bias correction process more transparent and easier to interpret. Also, these probabilities are only used during the question selection stage, and not treated as item parameters that can be directly estimated. Our current approach aims to capture general behavioral trends, for example, $\pi_s$ reflects an overall tendency for careless mistakes across the population. We agree that adapting or learning these probabilities dynamically could improve the model's realism. In future work, we plan to explore strategies that allow us to optimize $\pi_g$ and $\pi_s$ jointly during the question selection process.

Q4: Eq. (5) is derived by performing Taylor expansion on Eq. (4), where the authors introduce a complement set $Z$. Why not directly apply Taylor expansion to Eq. (3) instead?

A4: Yes, performing Taylor expansion directly on Eq. (3) is mathematically equivalent to the approach we use in Section 3.1 with the introduction of complement set $Z$. Our intention here was to convey: rather than physically removing the influence of $Z$ from the item pool, we apply a soft adjustment by down-weighting its contribution. This idea aligns with the strategy discussed later in Section 3.3, where we handle noisy samples in set $S$ by down-weighting them instead of removing/replacing them. By keeping this treatment consistent across sections, we aim to improve the overall conceptual coherence of the method. To avoid confusion, we will clarify the equivalence of these two more explicitly in the revised version.

Thank you again for your careful review. If anything remains unclear, we'd be happy to discuss it further.

Reference:

[1] Krause, Andreas, and Daniel Golovin. "Submodular function maximization." Tractability 3.71-104 (2014): 3.

[2] Sener, Ozan, and Silvio Savarese. "Active Learning for Convolutional Neural Networks: A Core-Set Approach." International Conference on Learning Representations. 2018.

Thanks for your positive feedback. If anything unclear, please feel free to contact us.

Hi reviewer W1GP!

Do you find the rebuttal convincing?

Thank you very much for your high evaluation of the quality of our paper. Below are our detailed responses to your concerns regarding the experiments:

Q1: Even considering the existing trends of the task, I have some doubts about whether it is a novel method that shows overwhelmingly good performance. I don't feel that it is a significant improvement over the existing baseline.

A1: This is a great point. In fact, almost no existing selection algorithm achieves overwhelmingly good performance in terms of ACC/AUC in adaptive testing. This is largely due to the nature of the task: these metrics are evaluated under small sample sizes (e.g., 5 or 10 items), but the theoretical upper bound (i.e., using all items) is limited by measurement model itself (e.g., IRT). For example, due to noise in student responses and the fitting capacity of IRT models, the AUC upper bound on the NIPS-EDU dataset is only around 0.79. Our method already achieves 0.78 with just 20 items, while the best of the other methods reaches only 0.76-0.77. So, further improvement is inherently constrained.

As you rightly pointed out, this suggests that ACC/AUC may be approaching saturation for this task. For example, UATS[1] only improved upon the previous best method by 0.5%–1%, and CCAT[2] did not outperform previous SOTA methods in terms of AUC/ACC at all. Because of this, the community often turns to another metric: ability estimation error (MSE) [3], which more directly aligns with the core goal of adaptive testing: minimizing estimation error. As shown in Figure 2 of our paper, our method achieves a 15% reduction in the number of items needed to reach the same estimation error, which we believe is a substantial and meaningful improvement.

Q2: It is questionable whether any comparisons with recent studies have been made. For example, we could easily find a comparable baseline, 'Liu, Zirui, et al. "Computerized adaptive testing via collaborative ranking." Advances in Neural Information Processing Systems 37 (2024): 95488-95514.'

A2: Yes. This is a recent and relevant work. Although we cited it in our paper, we didn't include it in the experimental comparison. The main reason is that its goal is quite different: it focuses on ranking students, while our work targets accurate ability estimation. Since their method is not optimized for estimation accuracy, we initially felt the comparison might not be entirely fair. That said, we agree it's important to be thorough. So, we've now included a comparison based on MSE across three datasets (ASSIST, NIPS-EDU, and EXAM), shown below:

[Table A – Dataset ASSIST]

[Table B – Dataset NIPS-EDU]

[Table C – Dataset EXAM]

As shown, our method consistently outperforms CCAT in terms of MSE across all steps and datasets. We'll include these results in the next version of the paper for completeness. If you have any other baselines you'd recommend for comparison, we’d be more than happy to include them as well. Thanks again for your valuable feedback!

Q3: Is there a better baseline or a dataset that we can experiment with? For example, we can think of JUNYI as a dataset.

A3: Thank you for your suggestion. We agree that exploring additional datasets can help further validate the effectiveness and generalizability of our approach. In response, we have conducted experiments on the JUNYI dataset, and the results are presented below. We plan to include these results in the next version of the paper.

Q4: However, in UAT and CCAT published at a similar time, there is a considerable difference in the baseline performance even though the same dataset and metrics are used. Why is this so?

A4: Thank you for raising this question. Yes, although UAT and CCAT were published around the same time and used the same dataset and evaluation metrics on the surface, there are notable differences in their baseline performance. This discrepancy can be attributed to several key factors:

• CCAT primarily focuses on ranking the examinees, and thus adopts Ranking Consistency metrics. In addition, the optimization strategies used in CCAT differ significantly: it employs MCMC-based methods and specific gradient descent, which are tailored to their specific modeling objectives.

• Data Splitting Strategies: The data partitioning schemes differ between the two works. In our study, we follow a 7:2:1 split for training, validation, and testing, while UAT adopts a 6:2:2 split. Such differences can impact the model’s generalization and reported performance.

• Data Preprocessing and Filtering: UAT is a data-driven method and applies its own data filtering criteria. Based on the released code and dataset details, it appears that UAT filters out a considerable number of examinees or items with insufficient interaction records. This can lead to a cleaner but less diverse dataset, which may inflate performance metrics.

Q5: If there can be differences in each experiment, it seems that sufficient experimental data is needed (such as mean or std).

A5: Yes, we completely agree. In our paper, Figure 2 already presents the mean and standard deviation of MSE, with results averaged over 10 repetitions and error bars indicating the standard deviation. To further enhance the transparency of our findings, we will include the std of AUC/ACC metrics in the appendix in the next version.

Please feel free to reach out if you have any further concerns or would like to discuss in more detail!

[1] Yu, Junhao, et al. "A unified adaptive testing system enabled by hierarchical structure search." Forty-first International Conference on Machine Learning. 2024.

[2] Liu, Zirui, et al. "Computerized adaptive testing via collaborative ranking." Advances in Neural Information Processing Systems 37 (2024): 95488-95514.

[3] Cheng, Ying. Computerized adaptive testing—new developments and applications. University of Illinois at Urbana-Champaign, 2008.

Thank you very much for the thoughtful follow-up and your encouraging words. You're right: when ACC/AUC approaches saturation, especially under short test lengths, it becomes increasingly difficult to showcase meaningful improvements. We truly appreciate you highlighting it. To address this, one approach we adopt in our paper is using Mean Squared Error (MSE) for ability estimation. As mentioned both in A1 and the main text, MSE directly reflects how accurately we estimate a student’s latent ability, which is the fundamental goal of adaptive testing. Compared to ACC/AUC, it tends to be more sensitive to improvements in selection strategies.

Beyond MSE, we believe there are several promising directions worth exploring:

• Designing more realistic evaluation settings, such as testing robustness and stability under perturbations. For example, in our paper, we experiment with noise injection (Table 5) to evaluate how well different methods hold up under uncertainty. This helps reveal the more fundamental strengths of each approach.

• Improving the capacity of the measurement model itself. As noted in A1, the theoretical performance ceiling is often limited by the expressiveness of the model (e.g., IRT). Developing more powerful or flexible measurement models could raise the upper bound of achievable AUC, even when using all available data. But, measurement modeling is itself a distinct research area, and advancing it requires broader community collaboration.

• Involving human experts in the evaluation loop. Though expensive, deploying systems in real or simulated environments with expert judgment can provide valuable insights into how meaningful and appropriate the selected questions are. Some recent works in personalized learning have taken steps in this direction.

These ideas are still under exploration and haven’t yet become standardized in the field, but we are committed to contributing toward building better evaluation benchmarks. We look forward to hearing more thoughts from you and please don’t hesitate to continue the discussion if you have further suggestions.

Thank you for your response. Let me be a little more specific about my question.

For example, from what I know about Active Learning (I think MAAT is one of the baselines), there is a certain trade-off between uncertainty and diversity. Accordingly, in Active Learning research, uncertainty-based methods record good performance, but they have the limitation that they may miss the diversity of the overall distribution. Of course Acc is also measured, but a diversity-based metric is presented to compare performance from various angles.

So my question is: Is there a way in Adaptive Testing to measure the similarity to the overall distribution or the diversity of the selected data? From what I've found, previous studies have presented performance in a similar way. Are there any reasons or limitations that make it so standardized?

Thank you so much for the follow-up. This is a really thoughtful point. Yes, in Active Learning, there's often a trade-off between uncertainty and diversity. However, we believe diversity hasn’t become a standard metric in Adaptive Testing because the goal here is different. In most cases, adaptive testing is focused on estimating a student’s latent ability as accurately and efficiently as possible. For that, information-based methods (like Fisher or KL) are more directly tied to reducing estimation error. These approaches are backed by well-established theory from psychometrics, where minimizing uncertainty in the estimate is the main objective [1].

Diversity, on the other hand, doesn’t always help with that. Here’s a simple example to illustrate:

Imagine you have a question pool with questions ranging from very easy to very hard, say difficulty levels from 0.0 to 1.0. Now, suppose you pick questions that are evenly spaced out to “cover” the whole range (that would look diverse). But let’s say one student’s ability is around 0.7. In that case, asking questions that are way too easy or way too hard isn’t very informative (waste of time). The most helpful questions are actually those close to their ability level, maybe between 0.6 and 0.8. So selecting based on uncertainty/informativeness, i.e., choosing questions where the model is most unsure actually leads to better estimation in practice. You can think of it a bit like binary search algorithm: we get the most value by focusing around the target iteratively.

So while diversity is definitely an interesting angle, it doesn’t align with the core measurement goal of adaptive testing (i.e., accurate ability parameter estimation). That’s probably why it hasn’t been widely used as a standard evaluation metric in this field. In contrast, metrics like MSE directly reflect the error in ability estimation, while AUC/ACC evaluate how well we can predict student responses (which indirectly indicate how accurate our ability estimates are) [1].

We really appreciate you raising this point and it's a valuable perspective. If you have more ideas, we’d love to hear them and keep the conversation going.

Reference:

[1] Liu, Qi, et al. "Survey of computerized adaptive testing: A machine learning perspective." arXiv preprint arXiv:2404.00712 (2024).

Thank you very much for your positive feedback on the motivation, methodology, and evaluation of our work. This is truly encouraging our commitment to further exploring this direction. We have carefully considered all your suggestions and have made following responses:

Q1: Table 1: To improve clarity, it would be helpful to explicitly label the variants of the proposed method in the table (e.g., "CFAT + IRT" and "CFAT + NeuralCDM") so that readers can easily distinguish the configurations being evaluated.

A1: Thank you for pointing this out. To avoid confusion and improve clarity, we will revise Table 1 in the next version by merging the two sub-tables and explicitly labeling all method combinations, such as "XXX + IRT" and "XXX + NeuralCDM". This labeling will make it easier for readers to distinguish the configurations and better understand the comparative performance of each method.

Q2: In the evaluation section, Figure 2 presents results for the ability estimation and computational efficiency tasks but includes only a subset of the baseline methods. Could the authors clarify the rationale for selecting these specific baselines, and why the others were omitted?

A2: In the evaluation section, we chose to include only a representative subset of baseline methods in Figure 2 due to space limitations. Specifically, we selected one widely recognized method from each category, following the taxonomy in [1]: Fisher for traditional information-based methods, MAAT for active learning, NCAT for data-driven approaches, BECAT for subset selection, and CFAT as our proposed method. Within each category, the runtime differences among methods are generally minor. For completeness, we've collected the full runtime statistics across all baselines as follows:

As shown, CFAT achieves a strong balance between efficiency and accuracy. It runs only slightly slower than Fisher, yet delivers SOTA performance in ability estimation. Based on your suggestion, we will include the complete runtime statistics in the appendix of the next version.

We truly appreciate your suggestion. Please feel free to share any further insights or suggestions you might have.

Reference:

[1] Liu, Qi, et al. "Survey of computerized adaptive testing: A machine learning perspective." arXiv preprint arXiv:2404.00712 (2024).

Thank you for your positive feedback on the motivation, core idea, and experimental results of our paper. Here are the responses to each of your questions:

Q1: ...How could we assume the Hessian matrix is invertible?

A1: We appreciate the opportunity to clarify. In the context of adaptive testing, this assumption is actually standard and reasonable. As mentioned in Section 3.1, line138 of our paper: "$\mathcal{H}(S, \theta)$ is invertible, which holds under standard regularity conditions in IRT. For complex neural network-based models, where computing the exact Hessian is often intractable, we adopt a quasi-Newton approximation (details are provided in Appendix A)."

Let us explain this in more detail:

• The invertibility of the Hessian is a widely accepted assumption in adaptive testing, often referred to in psychometric literature as "identifiability and estimability under local independence and monotonicity assumptions" [1]. In fact, under standard regularity conditions of IRT models, the loss function $\ell(\theta)$ is strongly convex [2][3], which ensures that the Hessian $\mathcal{H}$ is positive definite and thus invertible. So in classical IRT settings, this assumption is well-supported both theoretically and empirically. More importantly, all large-scale standardized tests today (e.g., GRE) are fundamentally based on IRT, which further motivates the paper’s focus on IRT-based frameworks.

• For more complex models like neural network approaches (e.g., NeuralCDM), computing the exact Hessian is often intractable due to the high dimensionality and nonlinearity of the parameter space. Therefore, in these cases, we adopt a quasi-Newton approximation of $\mathcal{H}^{-1}$, as detailed in Appendix A. This allows us to iteratively approximate the inverse Hessian without computing it directly, which makes the approach computationally feasible while still preserving the theoretical foundation of using $\mathcal{H}^{-1}$ for subsequent analysis.

To avoid confusion, we'll make this assumption and its justification more explicit in the revised version. We hope this could address your concern.

Q2: The datasets used in the experiment section are not well introduced, e.g., statistics, distribution.

A2: Thank you for pointing this out. Below is a summary of the key statistics for the datasets used in our experiments:

We will also provide additional visual analyses of the datasets (e.g., the distribution of the number of interactions per student), which are difficult to present here. We apologize for the oversight and all detailed dataset information and visualizations will be included in the appendix in the revised version.

Q3: It's not clear how the method can be used in large scale applications.

A3: That’s a great question. Our method is actually well-suited for large-scale applications for two main reasons:

1. No training required: Our approach is based on a simple greedy algorithm and does not involve any model training. This makes it significantly more efficient than training-heavy methods like reinforcement learning or active learning. As shown in Figure 2(b), our method achieves at least 12× speedup in runtime.

2. Validated on large-scale datasets: We've tested our method on very large datasets (Table 1), such as NIPS-EDU, which includes over 220,000 students and 19 million response logs. Moreover, our method remains robust even under high-noise conditions in realistic testing environments (Table 2).

We appreciate your suggestion and we will highlight these advantages more clearly in the conclusion section in the next version, especially in the context of large-scale deployment. If anything's unclear, we'd be happy to discuss them.

Reference:

[1] Lord, Frederic M. Applications of item response theory to practical testing problems. Routledge, 2012.

[2] Zhuang, Yan, et al. "A bounded ability estimation for computerized adaptive testing." Advances in Neural Information Processing Systems 36 (2023): 2381-2402.

[3] Yu, Junhao, et al. "A unified adaptive testing system enabled by hierarchical structure search." Forty-first International Conference on Machine Learning. 2024.

Thank you very much for the insightful follow-up. I fully understand your concern — there does appear to be a potential tension between the low-rank nature of the task (i.e., selecting a small number of informative questions) and the assumption of Hessian invertibility. We’d like to clarify that while our goal is to select a compact subset, this does not imply that the Hessian of loss wrt $\theta$ is low-rank: For simpler models like IRT, the invertibility of the Hessian is well-established under standard regularity conditions such as local independence and monotonicity. This theoretical foundation supports its use in many information-based strategies. For more complex models (especially neural architectures), invertibility is harder to guarantee, as you rightly pointed out. That’s why, as mentioned in A1, we adopt quasi-Newton approximations of the inverse Hessian in our paper, which provide a computationally feasible alternative while preserving the core principle.

Moreover, even when the Hessian becomes ill-conditioned or non-invertible (e.g., when very few items are selected using complex NN model), our method still acts as a heuristic information-based strategy. It can be viewed as a greedy algorithm that iteratively selects items contributing the most to the information function, even if the theoretical foundation of invertibility is no longer strictly valid at early stages. The experimental results in this paper can prove this. We’ll make sure to clarify this assumption more explicitly in the revised manuscript. Your comment has been very helpful.

As for your second question about the EXAM dataset (Avg. 9.27 responses/student) and the reported results at @10 and @20:

• While the average number of responses is 9.27, this figure includes a long tail of students with very sparse responses. In our experiments, we filter out students who have fewer than the required number of observed responses when evaluating @10 and @20 metrics (like previous works). This ensures that the reported results are based on sufficient data and are meaningful for comparison across methods.

• Additionally, the EXAM dataset is commonly used in prior adaptive testing studies precisely because it reflects a real-world low-resource scenario: many students only complete one test session, typically containing 10–20 questions. Evaluating performance on such sparse data is still important, particularly for metrics like @5, which remains reliable even under limited observations. We’ll make sure to clarify this preprocessing step in the revision to avoid any potential confusion.

Thank you again for your thoughtful and constructive questions, and they’ve helped improve the clarity and rigor of our work. Please feel free to continue the discussion if anything remains unclear.

Thanks for your thoughtful comment, and we appreciate how deeply you've engaged with the paper. YES. Bayesian methods are a very natural fit for adaptive testing, and there’s actually a fair amount of traditional psychometric work using Bayesian ideas [1]. However, in many existing papers [2–5], these approaches aren’t typically included as baselines. There are a couple of practical reasons for that:

1. Most Bayesian methods are still built on top of traditional information-based strategies (like Fisher or KL baseline in our paper). They just incorporate uncertainty modeling over the ability parameter $\theta$. Because of this, prior works often treat Fisher and KL as representative of the broader class of information-based methods.

2. More importantly, Bayesian methods can be much more computationally expensive. Many of them require integrating over the posterior (sometimes even multiple times per question), which makes them less suitable for real-time applications.

But we really appreciate your suggestion, and we agree that it will be helpful to include those variants. Thus, we ran additional experiments on the ASSIST dataset with two classic Bayesian selection strategies:

• Bayesian Fisher: After each response, we update the posterior over ability as $p(\theta \mid \text{responses}) \propto p(\theta) \prod_{i \in S} p(y_i \mid q_i, \theta)$, and then select the question with the highest expected Fisher information $I$:

  $$q^* = \arg\max_q \int I(q, \theta)\, p(\theta \mid \text{responses}) d\theta$$

• Bayesian KL: This method chooses the question that leads to the greatest expected change in the posterior distribution:

Below are the MSE results across different test lengths:

And here’s a rough comparison of the time cost per question:

Due to time constraints in rebuttal, we’ve only tested this on ASSIST so far. As you can see, the Bayesian variants do offer slight improvements over their non-Bayesian counterparts, especially in early test stages. As the test progresses, there’s still a gap between these and more data-driven methods like UATS, and the computational cost is higher. However, including these Bayesian baselines does make the experimental comparison more complete, and we appreciate the suggestion.

Again, thank you for being such a responsible AC. Your suggestion is very valuable and we’ll be sure to include these comparisons in our revised version. If anything here is unclear or you have more ideas to share, we’d really love to keep the conversation going.

Reference:

[1] Liu, Qi, et al. "Survey of computerized adaptive testing: A machine learning perspective." arXiv preprint arXiv:2404.00712 (2024).

[2] Wang, Hangyu, et al. "GMOCAT: A graph-enhanced multi-objective method for computerized adaptive testing." KDD 2023.

[3] Liu, Zirui, et al. "Computerized adaptive testing via collaborative ranking." NeurIPS 2024.

[4] Zhuang, Yan, et al. "A bounded ability estimation for computerized adaptive testing." NeurIPS 2023.

[5] Yu, Junhao, et al. "A unified adaptive testing system enabled by hierarchical structure search." ICML 2024.