Predictive, scalable and interpretable knowledge tracing on structured domains

We thank the reviewer for their feedback and address their questions below.

accuracy of 55-80 is not a strong result

We believe there is a misunderstanding. Table 2 reports predictive accuracies on several tasks, and we only obtain accuracies around 55% for between-learner generalization on Assist17. This task is deliberately designed to be difficult since the lack of historical data prevents learner personalization (note that baselines perform even worse). Accuracies for within-learner predictions (most commonly reported in related work), are considerably higher (63% to 83%), outperforming all baselines (incl. GKT and QIKT added based on reviewer feedback).

We realize it may not be obvious why higher predictive accuracies are not common in KT, which is known to be difficult due to selection bias: data are collected from learning systems designed to engage learners with KCs that are at the edge of their abilities. Thus, the data collection process inadvertently focuses on learner/KC-pairs where performance is particularly hard to predict.

more details on the datasets, particularly from the perspective of diversity. Claims about educational effectiveness and knowledge graphs that do no reflect a sufficient cross section are suspect at best and can be actively harmful.

We appreciate these important concerns. We address them using more cautious language in the introduction, and more details on our datasets and the selection criteria in the Appendix.

While we expect our model to enhance KT in structured domains beyond mathematics, we also want to avoid implying, even indirectly, that our model is universally applicable. Accordingly we propose the following edits to the introduction (removed, added): “[…] we present PSI-KT, a novel approach for inferring interpretable learner-specific cognitive traits and a shared knowledge graph of concept prerequisites relationships. We demonstrate the effectiveness of our approach on three real-world educational datasets covering structured domains, where our model outperforms existing baselines […]”.

We will clarify our choice of datasets in A.3.2. In brief, we require datasets in (1) structured domains with (2) KC labels and (3) high temporal resolution. These conditions rule out Statics2011 (2), Assist09 and Assist15 (3) and Junyi20 (3).

• Strengths: The primary significance of these results is in the interpretability of the results.
• Interpretability and scalability were not well evaluated and much of that was in the form of "correct by construction".

We are glad that the reviewer acknowledges the significance of the interpretability. A strength of our generative model is that we can explicitly define interpretable dependencies. However, far from assuming the interpretability is “correct by construction”, we spend the entire Sec. 4.3 to critically test this hypothesis using multiple metrics (specificity, consistency, disentanglement, and operational interpretability). Here, the first three are information theoretical metrics, which are agnostic to any preconceived notion of interpretability precisely to avoid the risk of assuming “correct by construction”. We also analyze the interpretability of the inferred graph (see next question).

While we appreciate suggestions for further interpretability analysis, we believe the interpretability of our model is very carefully evaluated and is one of our most significant results (as acknowledged by the reviewer). Given the diverse interpretability metrics that were carefully chosen to avoid unfairly favoring our model, we find it remarkable that our model is consistently superior than baselines (Tables 3-5).

We apologize for any confusion regarding scalability. In educational contexts, we specifically care about scalability where data from each learner increases over time, and aim to maintain efficient retraining and robust prediction accuracy as new data emerges. Figure 3 shows our model excels in both aspects, making it a compelling choice in future educational settings.

The prerequisite graph was interesting, although the correctness of the graph was not well quantified.

We are grateful that the reviewer is interested in our prerequisite graph. To evaluate inferred graphs (across now 7 models), we tested graph alignment against human-annotated prerequisites (both expert and crowd-sourced) with 3 metrics, and graph operational interpretability using Bayesian causal analysis to predict future learning outcomes from inferred edges, with favorable results (Table 5, and Appendix Fig. 12).

Additionally, given suggestions from reviewers (J4qa) and (rAUs), we now support these quantifications with an ablation study that shows our inferred graph is a key ingredient for our model to achieve its superior predictive accuracy (Appendix Fig. 13 and Table 16). We remain open for additional analysis to improve the quantification of the correctness of the graph and welcome your suggestions.

Thank you, authors, for all the clarifications, updates, and changes. I still think that the strongest parts of this paper are in the interpretability of the work and the knowledge graph, but after clarification. I have updated my review based on your revisions and I hope to see this work and any follow on papers in publication soon.

We thank the reviewer for their valuable feedback, which has allowed us to improve the manuscript by including additional experiments with new baselines and more data, and also adding clarifications on methodology, interpretability, and case studies.

Q1: Experiments on more learners

Why did the authors choose to experiment with only a limited portion of the datasets? The explanation provided, "to simulate real-world data constraints in education," may benefit from additional clarification.

We thank the reviewer for spotting that “real-world data constraints” is imprecise and leads to confusion. We now (a) clarify our focus on small-to-moderate cohort sizes and (b) extend performance evaluations to all learners. In detail:

1. We reformulated the start of Section 4.1:
   “In our evaluations, we mainly focus on prediction and generalization when training on 10 interactions from up to 1000 learners. Good KT performance with little data is key in practical ITS to minimize the number of learners on an experimental treatment (principle of equipoise, similar to medical research [1]), to mitigate the cold-start problem, and extend the usefulness of the model to classroom-size groups. To provide ITS with a basis for adaptive guidance and long-term learner assessment, we always predict the 10 next interactions.”
2. We have run additional experiments and now report performance for the maximum number of learners available in each dataset after preprocessing (Figure 2, Table 11). We discuss them in Section 4.1:
   “Figure 2 shows that PSI-KT's within-learner prediction performance is robustly above baselines for all but the largest cohort size (>60k learners, Junyi15), where all deep learning models perform similarly.
   The between-learner generalization accuracy of these models when tested on 100 out-of-sample learners is shown in Table 2, where fine tuning indicates that parameters were updated using (10-point) learning histories from the unseen learners. PSI-KTshows overall superior generalization except on Junyi15 (when fine-tuning).”

We are grateful for the opportunity to clarify our focus on the < 1000 learners setting, and for encouraging us to look beyond it into large cohorts. The differences in behavior between models warrant further research, which we have reflected in the discussion:
“We further find that PSI-KT has remarkable predictive performance when trained on small cohorts whereas baselines require training data from at least 60k learners to reach similar performance. An open question for future KT research is how to combine PSI-KT's unique continual learning and interpretability properties with performance that grows beyond this extreme regime.”

Q2: QIKT baseline

The introduction of interpretable KT methods is not comprehensive. For instance, recent approaches like IKT, ICKT, and QIKT incorporate interpretable psychological and cognitive modules into their methods. Could the authors consider using more recent interpretable deep learning methods like QIKT as their baseline comparisons? Doing so could enhance the credibility of the study.

We thank the reviewer for the valuable feedback and pointing us towards additional baselines. We found that QIKT is an excellent baseline to add (together with GKT [5], proposed by reviewer J4qa) and are pleased that our more comprehensive assessment (Figure 2-3, 12 and Table 2, 9-13, 15) provides even stronger support for our contributions.

We have improved the related work (Sec. 2), now covering interpretable KT approaches IKT [2], IEKT [3] and QIKT [4].

Adding even more baselines might still be valuable, but we were unable to include more because (1) the official IKT code lacks a key component (tree structure) making a fair comparison challenging; (2) while we were unfortunately unable to find a reference for ICKT, we found the similarly named IEKT and now reference it in the related work. We did not add it to evaluations as we believe QIKT provides a superior baseline. Should ICKT differ from IEKT, we welcome a reference for potential future inclusion.

We are grateful for the strong evaluation and detailed feedback on our analyses.

Could the authors provide some explanations about the four dimensions of cognitive traits and how they represent specific characteristics of students? It would be particularly helpful if these dimensions can be correlated with concepts from cognitive science. Additionally, I'm interested in an experimental analysis of the impact of the other two dimensions.

We thank the reviewer for pointing out potential confusion with the interpretation of the cognitive traits. We have now revised the methods section to more clearly explain these dimensions (Sec. 3.1):
“Specifically, $\alpha^\ell$ represents the forgetting rate [1,2] $\mu^\ell$ (via $\tilde{\mu}^{\ell,n}_k$) captures long-term memory consolidation [3] for practiced KCs and expected performance for novel KCs, $\sigma^\ell$ is knowledge volatility, and $\gamma^\ell$ indicates transfer ability [4] from performance on prerequisite KCs.”

Following the suggestion, we have performed additional interpretability experiments that relate the other two dimensions (transfer ability and knowledge volatility), to appropriate behavioral measures (Fig. 10 in Appendix A.6.4). In brief, we relate transfer ability to successful transitions on the prerequisite graph and knowledge volatility to temporal variability in individual performance. In all cases, our model achieves the highest match between parameters and behavior. We believe these additions have substantially improved the interpretability of the cognitive traits captured by our model.

Have the authors considered supplementing with essential ablation study and adding baseline models that explicitly take into account the knowledge graph structure, such as GKT[1] or SKT[2], the latter of which also considers prerequisite relationship between concepts?

We are grateful for the suggestion of an ablation analysis (also mentioned by reviewer rAUs) and the recommendation of additional baselines. The suggestion of performing ablation analyses was incredibly helpful, and allowed us to discover that all components of the model are crucial for performance. Additionally, the addition of a strong structure-aware baseline (GKT) has strengthened the credibility and thoroughness of support for our model.

In a new ablation analysis (Table 16 and Figure 13, Appendix A.8), we ablate the graph structure, the individual traits, and learner dynamics, observing significant drops in accuracy in each case. No single ablation reliably corresponds to the largest drop in accuracy across datasets. Thus, we believe that all components make important contributions to the robustness and generalizability of our model.

We have also added GKT to our set of baselines (along with an additional model QIKT [5] mentioned by other reviewer k3Pn). GKT represents one of the strongest baselines and expands our comprehensive assessment of alternative models (Figure 2-3, 12 and Table 2, 9-13, 15), which altogether provides even stronger support for our model.

While SKT has a number of innovative methods for modeling both similarity and prerequisite structures, we could not find how to perform data splitting to calculate the graph (either in the code base or paper). Specifically, the graph is calculated from performance statistics (i.e., correct/incorrect counts), which cannot be directly used to predict the same performance data (to avoid circularity). This same circularity is not an issue when the graph inference is part of an end-to-end learning pipeline, as in ours and other models. For this reason a fair comparison seems challenging.

In sum, we have clarified the interpretability of cognitive traits, added additional analyses for the other two dimensions, performed an ablation analysis, and added an additional model to our baselines. We believe this has greatly strengthened the paper and welcome the reviewer to consider revising their score to reflect these changes.

References:

[1] Ebbinghaus, H. (1885). Über das gedächtnis: untersuchungen zur experimentellen psychologie. Duncker & Humblot.

[2] Averell, L., & Heathcote, A. (2011). The form of the forgetting curve and the fate of memories. Journal of mathematical psychology, 55(1), 25-35.

[3] Meeter, M., & Murre, J. M. (2004). Consolidation of long-term memory: evidence and alternatives. Psychological Bulletin, 130(6), 843.

[4] Bassett, D. S., & Mattar, M. G. (2017). A network neuroscience of human learning: potential to inform quantitative theories of brain and behavior. Trends in cognitive sciences, 21(4), 250-264.

[5] Chen, J., Liu, Z., Huang, S., Liu, Q., & Luo, W. (2023). Improving interpretability of deep sequential knowledge tracing models with question-centric cognitive representations. arXiv preprint arXiv:2302.06885.

We thank the reviewer for the valuable feedback. We address them below and also increase the manuscript by adding more clarifications and new ablation studies.

Could you provide insights into the dataset limitations and discuss potential challenges in applying the model to other educational domains or datasets?

We thank the reviewer for the thorough feedback on our paper. We agree that the representativeness of empirical evaluations should be a central concern for ours and other models, and now highlight it in the discussion:
​​“Although we designed PSI-KT with general structured domains in mind, our empirical evaluations were limited to mathematics learning by dataset availability. We highlight the need for more diverse datasets for structured KT research to strengthen representativeness in ecologically valid contexts.”

We expand on limitations in Appendix A.3.2 where we now describe in detail the datasets and the criteria we used to select them (also in response to reviewer da66). In brief, our choice of datasets was constrained by the following requirements: we require data sets in structured domains with (1) KC labels and (2) high temporal resolution. These conditions rule out Statics2011 (1), Assist09 and Assist15 (2), and Junyi20 (2).

• Long-term retention modeling could be enhanced. The current exponential decay may be simplistic. Exploring more complex forgetting functions based on memory research literature could improve long-term predictions.
• Have you considered exploring more complex forgetting functions based on memory research literature to improve long-term predictions?

The reviewer is right that exponential forgetting alone, while common in the psychophysics [1] and recent KT literature [2], may be simplistic [3]. That is the reason why we extend the vanilla Ornstein-Uhlenbeck process with a reversion mean for long-term retention (now better clarified in Sec 3.1) [3], and parameterize the process with learner- and time-dependent cognitive traits, resulting in a very expressive model of memory dynamics.

Despite our good empirical results with this extended OU model (see also new ablation results below), we do agree with the reviewer that supporting a variety of forgetting functions is key to bringing the benefits of memory research to KT (and conversely), a concern we now reflect in the discussion:
“We use an analytically marginalizable OU process for knowledge states in PSI-KT, resulting in an exponential forgetting law, similar to most recent KT literature. Future work should support ongoing debates in cognition by offering alternative modeling choices for memory decay e.g., power-law [4], thus facilitating empirical studies at scale. “

Because the scalability properties of stochastic process capturing alternative forgetting functions (e.g., power law) require additional investigation, we intend to address this in future work.

Could you perform ablation studies to dissect the contributions of different model components to predictive accuracy, providing insights into the model's strengths?

We are very grateful to the reviewer for the suggestion of performing ablation studies, which was suggested by the reviewer J4qa as well. When we ablate any of the graph structure, the individual traits, and learner dynamics, we observe significant drops in accuracy across all three datasets. No single ablation reliably corresponds to the largest drop in accuracy across datasets. Thus, we believe that all components together allow our model to adapt to a variety of educational contexts and datasets, thereby enhancing its robustness and generalizability.

We present the results and their interpretation in Appendix A.8 (Table 16 and Figure 13) and summarize them in Section 4.1:
"The advantage of PSI-KT comes from its combined modeling of KC prerequisite relations and individual learner traits that evolve in time."

Overall, we are encouraged that these insightful suggestions resulted in clarifications and analyses that strengthen the relevance of our contribution, helping us to reach out beyond the machine learning community. We hope that the reviewer will consider revising our score and we welcome further suggestions for improvement.

References:

[1] Ebbinghaus, H. (1885). Über das gedächtnis: untersuchungen zur experimentellen psychologie. Duncker & Humblot.

[2] Nagatani, K., Zhang, Q., Sato, M., Chen, Y. Y., Chen, F., & Ohkuma, T. (2019, May). Augmenting knowledge tracing by considering forgetting behavior. In The world wide web conference (pp. 3101-3107).

[3] Averell, L., & Heathcote, A. (2011). The form of the forgetting curve and the fate of memories. Journal of mathematical psychology, 55(1), 25-35.

[4] Wixted, J. T., & Ebbesen, E. B. (1997). Genuine power curves in forgetting: A quantitative analysis of individual subject forgetting functions. Memory & cognition, 25, 731-739.