Cuff-KT: Tackling Learners' Real-time Learning Pattern Adjustment via Tuning-Free Knowledge State-Guided Model Updating

In the final step of the framework, expert analysis is still required. This feels a bit contradictory to the goal of improving automation. Since you're already using LLMs, why not go a step further and design an evaluation agent to replace or supplement the expert analysis? This could boost efficiency and reduce dependence on experts.

While the framework diagram is clear, I think it could be made even more detailed. For example, showing the flow of data between different modules could make it easier to understand how the system operates, especially for readers unfamiliar with the field.

The paper discusses two key components of TestAgent: adaptive question selection and cognitive diagnosis. I think an ablation study would be helpful to evaluate the individual contributions of these components. It could also be interesting to see what happens if human experts take over one of these components, providing insight into human-machine collaboration.

The appendix gives some examples of the tests, but it doesn’t show the full prompts sent to the agent. For an LLM-based system, the design of the prompts is crucial, as it defines the task and the agent's role. I suggest the authors provide more detail about how these prompts are structured to give a clearer picture of how the system operates.

1. My biggest concern is why the paper introduces the topic from the perspective of distribution shift and compares with anomaly detection methods. How are these two concepts related? My understanding is that it’s one of the causes of distribution shift. The term "anomaly detection" first appears in line 377. What constitutes an anomaly in knowledge tracing? After reading the motivation study, I speculate that anomaly detection in knowledge tracing might refer to changes in students' accuracy rates. In summary, the relationship of two concepts could benefit from more explanation and clarification in the introduction and related work.
2. If this is indeed anomaly detection, then I believe the authors should compare their method with HD-KT[1] to validate its effectiveness.
3. The author cited numerous papers in the related work, such as iAKT [2], which address handling different distributions. Why weren’t these methods included for comparison?

[1] Ma, Haiping, et al. "HD-KT: Advancing Robust Knowledge Tracing via Anomalous Learning Interaction Detection." Proceedings of the ACM on Web Conference 2024. 2024.

[2] Wong, Cheryl Sze Yin, et al. "Incremental context aware attentive knowledge tracing." ICASSP 2022

The literature review omits recent models like QIKT, SimpleKT, and PSIKT. Including these would clarify Cuff-KT’s positioning. Additionally, the rationale for selecting the three evaluation baselines needs more explanation.

The clarifications of the experiment setup and details of the methodology need improvement, but do correct me if I am missing anything here:

1. In lines 226 and 431, the authors mention “e.g., KL-divergence,” but it is unclear if other distance measures were considered. It would also be helpful to know how the distribution over embeddings was approximated—was it, for instance, fitted to a Gaussian?
2. In the controller setting, is there any specific reason that the authors chose to compare step $k$ and intermediate time-step $\frac{k}{2}$, i.e., why not using a sliding window setting? Theoretically, as the time step $k$ goes up, the discrepency between the two distributions will not decrease. This means the $\mathrm{KL}^j$ in Eq.4 will always become larger. Does the model prioritize students with longer learning histories, thereby increasing their likelihood of selection?
3. In Sec. 3.2.2, I would need much more clarifications so that I can comment on this part. In line 265, $c_i$ and $r_i$ are input for question difficulty and learner ability. Why do authors choose $c_i$ as the input or it is a typo? This question applies to all of the question/concept notations in this section, e.g., “embedding the questions $c_{1: k}$”. I will try to comment on the soundness of generator again after the authors clarify this section. Meanwhile, how do authors embed the time information, i.e., what is $t_{1:k}$ in Eq. 5?
4. The experiment training and validation needs more clarifications. In lines 368-370, the authors say the historical interactions are split into training, validation, and test sets based on the timestamps and groups. I am not clear how inter- and intra- evaluations are done and how the training and test work, could the authors provide better explanations (more details can come in the appendix if it exceeds page limit)?
5. Do the authors provide the effectiveness of the controller? How do the performance change if you remove the controller at all, i.e., the model will generate the parameters for every student? I also wonder the statistics of the student chosen by the controller, how many students are chosen in general and what is the correlation between the chosen students and their interaction length? As I wonder in question 2, if the length plays a dominant role (as the authors show in Fig. 7), what else is determining the chosen probability intuitively?

1. Some mistakes regarding presentation:

a. In Line 251 $ZPO^j$ should be $ZPD^j$ (The same is true in the source code train.py);

b. In Line 328, $k$ indicates the number of samples, but it has been defined in Line 265 (time-step $k$);

c. The use of notation in the paper is confusing, lacking a clear distinction between matrices, vectors and scalars, e.g., Line 274 and Line 313.

d. The counter of equations does not work well. For example, there are two Eq. (4) in page 5.

e. The citation format is inconsistent; for example, the conference paper lacks the address.

2. The task RLPA seems like the specification of the distribution shift in time series forecasting in knowledge tracing scenarios [1]. It’s believed that temporal distribution shifts are quite common in time series forecasting and has many solutions. However, the paper does not mention anyone in related work or make any of them compared with the proposed method.

3. Line 204 is unrelated to the proposed RLPA task. Minimizing this KL divergence is the same as the BCE loss in Line 328, which is the typical KT task loss. This formula doesn't hold any practical significance here.

4. The paper doesn’t clearly explain the experimental setup. For example, it’s hard to find a straightforward description of how datasets are divided in settings of inter-learner and intra-learner respectively; you have to look at the code to figure it out.

References:

[1] Fan et al. Dish-TS: A General Paradigm for Alleviating Distribution Shift in Time Series Forecasting. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, Washington, DC.