Learning Skill-level Student Abilities with Item Response Theory

Writing: the writing needs substantial improvement for better flow and clarity. Key issues include 1) The abstract is unclear and fails to highlight the main motivations and contributions. Phrases like "a question difficulty network for learning question difficulties" are redundant; 2) The introduction provides too much detail about IRT without first explaining the model. For example, the discussion about the "question discrimination parameter" (lines 89-91) lacks necessary context, making it hard to understand the motivation; 3) While the methodology section is detailed, the formulas (Eq. 3-6) need more intuitive explanations and better formatting. Instead of presenting formulas which only show the network architecture, the rationale behind design choices should be clarified. Also, the notations need more structured explanations (duplicates in line 266 and 270).

Position of the paper: While the authors acknowledge existing IRT-based models in knowledge tracing, this work should also position itself among other interpretable KT models. I recommend providing more detailed comparisons in the literature review[1,2], particularly explaining why the focus is on IRT. Additionally, as noted in the limitations, the model does not account for student abilities, which is a crucial aspect of personalized education—especially given that the work claims such an application.

Understanding the IRF parameters: two of the contributions are learning skill-level student abilities and consistent IRT outputs. But the claim of student skill-level abilities should be demonstrated while in the current experiments there is no explanation. Also, the statistics (mean, min, max) of the question-level parameters are not convincing.

1. Regarding the question-level parameters:

• I do not follow the purpose of Table 4 and the bold numbers. Why do authors claim that the student abilities and the question difficulties are not in the same range?
• It is not informative when authors aggregate over all students if you want to do personalized education. I am curious about individual student’s abilities and the difficulties of the questions they encountered.
• In Table 5, where do the ground-truth difficulties come from? Also, since the model regularizes the inferred parameters to some prior knowledge (Eq. 8 and 11). My questions are two folds:
  • How would the ablation study looks like if one relagurizer is removed at once? Also, what would be the experiment results if the empirical estimations $\hat{b}_q$ and $a = 1$ are used?
  • How would the empirical difficulty $\hat{b}_q$ correlates with the ground-truth? If it is correlated well, then the inferred difficulty parameters are not that useful.

2. Regarding the skill-level student abilities: There is no analysis of these inferred latent variables for each student and each skill. The SOTA performance in predicting performance can not claim “The skill-level student abilities and other IRT parameters generated by our model can be easily used for skill-meter building and other downstream tasks like adaptive learning and personalized recommendations. “ I recommend authors provide statistical analysis for the inferred student abilities 1) are useful for personalized recommendations. E.g., show the inferred parameters are specific to students [3]; 2) are truly skill-level abilities. e.g., correlation between these ability parameters and held-out student performance.

Lack of specific comparison and explanation for choosing LSTM: In the model’s embedding layer, the authors use LSTM to model students' historical sequences but do not explain the reason for this choice or whether they considered other potential architectures, such as Transformers or CNNs, as alternatives. While LSTM performs well for sequence modeling, it may have efficiency issues with long sequences or large-scale data. To strengthen the justification for their design, the authors could include comparisons between LSTM and other sequence models (such as Transformers or CNNs) in the experiments.

Lack of validation for the effectiveness of skill-level modeling: Although the authors propose an innovative approach to model student knowledge states at the skill level, the experimental validation primarily focuses on overall performance and parameter consistency. There is insufficient validation of the actual impact of skill-level modeling itself. The ablation study shows only the effect of different loss terms on the overall AUC, without providing specific insights into how well the model captures student abilities at the skill level. To further evaluate the model’s effectiveness, the authors could consider including a fine-grained analysis, such as examining the distribution of student abilities across different skills and their dynamic changes over time.

Limited discussion on the independent effects of the auxiliary task and regularization terms: The authors introduce a relationship loss as an auxiliary task to capture the relationships between questions and skills, enhancing skill embeddings to better represent question information. However, the importance and necessity of this auxiliary task lack theoretical explanation. In the ablation study, the authors explore the impact of the relationship loss and regularization terms on the model’s overall AUC, but this evaluation focuses on general effects across different datasets without analyzing the independent contributions of each loss term at the skill level or regarding IRT parameter consistency. The authors could conduct more detailed experiments by selectively removing or retaining each loss term, especially providing finer-grained comparisons at the skill level and in terms of IRT parameter alignment.

Some requirements of IRT do not make sense to me. "$\theta$ and $b_q$ should be on the same continuum" Is it really a requirement? Because both are scalar anyway so both of them are in $\mathbb{R}$. Similarly, is 4 really a requirement?

Perhaps the authors should compare to psiKT who was presented at ICLR 2024.

Zhou, Hanqi, et al. "Predictive, scalable and interpretable knowledge tracing on structured domains." The Twelfth International Conference on Learning Representations.

The proposed approach SKKT-IRT is very close to this other work, Section 3.2.3.

Vie, Jill-Jênn, and Hisashi Kashima. "Deep Knowledge Tracing is an implicit dynamic multidimensional item response theory model." ICCE 2023-The 31st International Conference on Computers in Education. 2023.

Wilson et al. (2016) managed to match the performance of DKT with retrained IRT, which is clearly a very simple model.

Wilson, Kevin H., et al. "Back to the basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation." in Educational Data Mining 2016: 539.

If I compare to Gervet et al. 2020, I see that a logistic regression (Best LR) has a AUC of 0.831 on Algebra 2015 which outperforms the proposed approach SKKT-IRT.

Gervet, Theophile, et al. "When is deep learning the best approach to knowledge tracing?." Journal of Educational Data Mining 12.3 (2020): 31-54.

According to this other paper, the performance of IKT on Assistments 2009 is 0.797 (which would outperform SKKT-IRT) and its performance on Algebra 2005 is 0.851. According to this same paper the performance of AKT on Algebra 2005 is 0.845 which would already outperform ExecCAKT.

Minn, Sein, et al. "Interpretable knowledge tracing: Simple and efficient student modeling with causal relations." Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 11. 2022.

I also encourage the authors to read about logistic knowledge tracing.

Pavlik Jr, Philip Irvin, and Luke G. Eglington. "Automated search improves logistic knowledge tracing, surpassing deep learning in accuracy and explainability." Journal of Educational Data Mining 15.3 (2023): 58-86.

I don't think it's a very good paper especially on a topic related to knowledge tracing, and at least not up to what I would consider to be the standard of average ICLR acceptance.

1. The motivation of this paper is not novel, and work on exploring the knowledge tracing task from the perspective of modeling skill dimensions has become common.

2. The writing of this paper does not flow well, for example, the Abstract section is too cumbersome, and in fact the description from motivation to challenge is too trivial not cohesive enough and not logical.

3. The design of the model is also very simple, simply adding an IRT interaction layer to the KT model that uses LSTM as a base module, and I don't see any novel design to solve the problems mentioned in the motivation.

4. The authors did not do interesting case studies to show such interpretable results as well as analysis, which is what makes the paper unattractive.