Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modeling

We appreciate your careful reading and detailed feedback on our paper. We address your concerns below and please let us know if there are remaining questions or unclear points.

Weakness 1: Unconvincing motivation and limited methodological novelty.

Thank you for highlighting these concerns and for giving us the opportunity to provide a clearer explanation.

• Our motivation is well-founded and supported by data analysis and relevant literature, involving two aspects:

1. Motivation 1 (collaborative modeling): Introducing collaborative signals between learners in CD can enhance performance and interpretability. This is supported by social learning theories[1], data analysis (see experiments 1 and 2 in the next response to Weakness 2), and empirical effectiveness on learner sequences[2] and group modeling[3], which demonstrate the benefits of incorporating collaborative signals. However, most CD models focus on individual modeling and often neglect collaborative aspects. Thus, we incorporate collaborative signals in to CD is both convincing and meaningful.
2. Motivation 2 (disentanglement): Modeling both individual and collaborative learner representations following Motivation 1 increases complexity, which traditional single-vector representations may not adequately handle. Thus, disentanglement is naturally introduced to convert single-vector representations into multiple vectors, reducing dimensional correlation and enhancing modeling ability, inspired by [4].

• Mthodological novelty on key modules:

1. Disentangled Encoder (inner-learner view): Inspired by DCD[3], our encoder considers both vector-level and dimensional-level disentanglement, unlike DCD, which only considers the latter. This approach's effectiveness is empirically demonstrated through ablation studies.
2. Collaborative Modeling (inter-learner view): This design is unique as relevant work in CD is almost blank. We provide theoretical derivations to support the model's rationale.
3. Co-Disentanglement Decoder: Inspired by residual connections, our decoder effectively and simply aligns inner- and inter-learner perspectives. This alignment represents a novel exploration in this direction.

Our model's innovation is supported by both empirical and theoretical evidence.

Weakness 2: Concerns on interpretability.

Thank you for your attention to interpretability.

We would like to clarify that our interpretability analysis is detailed in a case study in Appendix E, rather than through visualizations. This example demonstrates how Coral infers each learner's future performance by referencing the cognitive states of similar learners, thereby providing educators with a basis for their inferences.

To provide more data-based conclusions, we conduct additional experiments on ASSIST:

1. We represent $M$ learners $U$’ performance on $N$ questions with a matrix $X = \\{x\_{u,i}\\}\_{M \times N}$, where $x\_{u,i}$ indicates whether a learner $u$ answered problem $i$ correctly (1), incorrectly (0), or did not attempt it (-1). We split this matrix into $X\_{\text{train}} = \\{x\_{u,i} \mid u\in U, i \in [1, N/2]\\}$ and $X\_{\text{test}} = \\{x\_{u,i} \mid u\in U,i \in [N/2+1, N]\\}$. For each learner $u$ in $X\_{\text{train}}$, we find the most similar learner $v$ by maximizing $\text{cosine}(X\_{u}, X\_{v})$, and then compute the absolute difference $|\Delta(u,v)|$ in their passing rates on $X\_{\text{test}}$.
2. Based on a trained Coral, we find the most similar learner $v$ for each learner $u$ via learned embedding cosine similarity. We then calculate the absolute difference $|\Delta(u,v)|$ in the accuracy of problem-solving between $u$ and $v$ on the test data.

Mean and variance of all $|\Delta(u,v)|$ are listed as:

Experiment 1 shows that learners with similar training data experiences tend to have similar performance on future practice, supporting our collaborative modeling motivation from the perspective of dataset statistic. Experiment 2 proves the effectiveness and interpretability of learner embeddings learned by Coral. Both experiments reinforce that similarity of learner embeddings by Coral can align with statistical data. We will make this insight clearer in the revised paper.

Weakness 3: Computational inefficiency.

Our paper provides 3 alternative efficient solutions in Appendix F.2 to improve Coral's computational efficiency. Although these solutions may not guarantee theoretical optimality, the experimental results demonstrate their effectiveness, which prove Coral can be used to some large-scale scenarios. For real-time response, co-disentangled cognitive state (line 249) can be precomputed and stored, with only retrieval needed during online computation. We will continue to optimize the model efficiency in future studies.

Weakness 4: Concerns on hyperparameter tuning complexity.

Coral requires minimal hyperparameter tuning. We adjust only two parameters, $\beta$ and $K$, with other parameters fixed based on prior experience. Experimental results in Fig3 (c) show $K$ stabilizes after reaching a threshold due to attention control. Thus, when tuning, we can start with a larger $K$ and gradually decrease it. For $\beta$, we recommend values of 0.25, 0.5, and 0.75. Typically, 3 to 4 iterations are sufficient to find satisfactory parameters.

Weakness 5: Add more baselines.

We conduct additional experiments with four new baselines. The experimental setups, results and analysis are given in Public Response (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66).

[1] Albert Bandura and Richard H Walters. Social learning theory. 1977.

[2] Improving knowledge tracing with collaborative information. WSDM'22.

[3] Homogeneous Cohort-Aware Group Cognitive Diagnosis: A Multi-grained Modeling Perspective. CIKM'23.

[4] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.

Dear Reviewer Ty5J,

We appreciate your careful reading and detailed feedback of our paper, and giving us the opportunity to answer your questions.

During the rebuttal phase, we have carefully addressed each of your concerns and are eager to receive your feedback for further discussion. We understand and appreciate the effort you put into reviewing the paper, so we have summarized our response for your quick reading:

1. Our motivation is well-supported by psychological theories, existing literature, and data analysis. Our model is the first exploration in the disentangled collaborative CD direction and is innovative. Please refer to the rebuttal for details.
2. There is a misunderstanding concerning our interpretability experiment, which is thoroughly addressed in the case study in Appendix E. Additionally, we further provide dataset-based conclusions in the rebuttal that answer your questions.
3. Thank you for your concerns regarding the computational efficiency limitation. We agree that this is a main bottleneck for our Coral model. To address this, our paper presents three preliminary strategies to improve efficiency, as detailed in Appendix F.2, and will focus on improving computational efficiency of Coral in future research. Additionally, we proactively acknowledge the computational efficiency limitations in the limitations section. According to the NeurIPS 2024 checklist guidelines, this proactive acknowledgment is considered a positive practice and should not be penalized.
4. Our experiments mainly require the tuning of two hyper-parameters ($\beta$ and $K$) with a fixed learning rate, which is not complex. The hyper-parameter tuning complexity is similar to DCD and RCD in baseline.
5. We add four additional baselines in the rebuttal.

Thank you again for your time and effort, and we will revise the paper following each of your suggestions. If you have any further questions or need clarification on any point in our response, please let us know. We will respond promptly.

Best,

Authors.

Dear reviewers,

Thank you for recognizing our work as well as our rebuttal. We are pleased to have addressed your concerns and will revise the paper according to your suggestions.

We are very grateful to you for your suggestions on interpretability experiments from the data statistic perspective to help us improve the quality of our papers!

Thank you!

Best,

Authors

We appreciate your careful reading and detailed discussion of our paper. We address your concerns below and please let us know if there are remaining questions or unclear points.

For the weaknesses:

Weakness 1: Explain the solution of collaborative relation construction in detail.

Thank you for your feedback and for giving us the opportunity to clarify our solution.

Constructing collaborative connections aims to find $K$ collaborative neighbors for each learner $u \in V$ and build the collaborative graph $G$ (see lines 11-31 in Algorithm 1). This process is formulated as $\max P(G|V,Z)$, where $V$ is the set of learners and $Z$ is the learner state vector. To achieve this, we iteratively find the $K$ optimal neighbors for each learner $u$ with the following initialization: $u$'s neighbor set initially includes only $u$, i.e., $r_{u}^{0} = {u}$, and the non-neighbor set consists of all other learners, i.e., $V_u = V \setminus {u}$. The iteration starts and repeats $K$ times until $u$ has $K$ neighbors. Each iteration includes the following steps:

Iteration (for $k = 1, 2, \ldots, K$):

• Compute context vector of $u$ using $(k-1)$ neighbors: $\text{Context}(u) = \sum_{v \in r_{u}^{k-1}} Z_v$
• Calculate similarity scores between $u$'s context and each non-neighbor learner $v$: $\text{score}_{u,v}$ for each $v \in V_u$
• Select the most similar learner as the $k$-th neighbor: $b_{u}^k = \arg\max_{v \in V_u} \text{score}_{u,v}$
• Update neighbor set: $r_{u}^{k} = r_{u}^{k-1} \cup \{b_{u}^k\}$
• Update non-neighbor set: $V_u = V_u \setminus \{b_{u}^k\}$

We sincerely hope our explanation resolves your concerns. We will include more examples in the revised version for clarity.

Weakness 2 and Weakness 3: Issue of balance between theoretical details and experiments.

We apologize for any confusion. To enhance readability, we included detailed proofs and algorithms in the appendix while summarizing key conclusions in the main text of the paper. For instance, the core construction method of the collaborative graph is summarized in lines 202-207 of the main text, with detailed derivations in Appendix B and algorithms in Appendix C. This structure ensures that our key conclusions are understandable without requiring a deep dive into the theoretical derivations, as acknowledged by reviewers vXdf, t9qg, bUz3, and UPTU.

We will simplify the theoretical symbols in the revised version and include more examples to improve clarity. Additionally, we will increase the size and spacing of Fig.3 and 4 in experimental section.

Weakness 4: Concerns on performance improvement.

Coral's performance improvements are validated through experiments across three different scenarios (line 296), not solely in Table 1. We sincerely encourage you to consider the model's performance across all scenarios to appreciate Coral's comprehensive enhancement.

Specifically, Table 1 lists results for the normal scenario (20% test data), where Coral demonstrates significant improvements on Junyi and ASSIST. The improvement on the NIPS202EC dataset is relatively modest, primarily because the NIPS202EC dataset is very dense (as shown in Table 3, with 367.5 problem-solving records per learner), leading to smaller differences between models. Additionally, Figures 2(a) and (b) illustrate significant performance improvements in scenarios with data sparsity and cold start, which are commonly encountered in real platforms, showing that Coral significantly outperforms the baselines.

For the questions:

Question 1: Concept disentanglement impacts and insufficient motivation for collaboration modeling.

• Concept disentanglement impacts: To answer your first question, we conduct experiments removing the disentanglement on concepts by representing each learner with a single $C$-dimensional vector instead of the original $C$ $d$-dimensional vectors, denoted as w/o VecDis. The following experimental results on NIPS2020EC prove the effectiveness of disentanglement. |Model|ACC|AUC|F1|RMSE| |-|-|-|-|-| |w/o VecDis|0.71523|0.78331|0.72667|0.43412| |Coral|0.71622|0.79103|0.72890|0.43200|
• Clarify the motivation:

Thank you for your attention to our motivation and for giving us the opportunity to clarify. In fact, our core motivation is to integrate collaborative signals into CD. Based on this motivation, disentangled representations are introduced to manage the increased complexity of modeling both individual and collaborative representations. Detailed explanations are as follows:

1. Motivation 1 (collaborative Modeling): Introducing collaborative signals between learners in CD models enhances both performance and interpretability. It is sufficiently supported by social learning theories [1], data analysis (see experiments 1 and 2 in the response to Weakness 2 of Reviewer Ty5J), and empirical effectiveness on learner sequences [2] and group modeling [3], demonstrating the benefits of incorporating collaborative signals.
2. Motivation 2 (disentanglement): To model both individual and collaborative learner representations following motivation 1, complexity increases. Traditional single-vector representations may not adequately capture this complexity. Thus, disentanglement is naturally introduced to convert single vector into multiple vectors, reducing dimensional correlation and enhancing modeling capability, inspired by [4].

Thank you again, and we sincerely hope our explanation resolves your concerns. We will make this insight clearer in a revised paper.

[1] Albert Bandura and Richard H Walters. Social learning theory. 1977.

[2] Improving knowledge tracing with collaborative information. WSDM'22.

[3] Homogeneous Cohort-Aware Group Cognitive Diagnosis: A Multi-grained Modeling Perspective. CIKM'23.

[4] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.

Thank you for the detailed and constructive feedback! We answer your comments and questions below. Please let us know if you have additional questions.

For the weaknesses:

Weakness 1: How to mitigate bias issues in establishing learner collaboration relationships with insufficient interaction data.

Thank you for raising this issue. Our variational encoder can mitigate this problem by magnifying subtle differences in the latent feature space. In fact, existing CD models that rely solely on student interaction data typically face this challenge[1].

Weakness 2: Add a graph-based baseline.

Thank you for pointing out the absence of a graph-based baseline, i.e., SCD. We have conducted learner performance prediction tasks using SCD on three datasets, as shown in our Public Response (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66). Experimental results prove the consistent superiority of Coral compared with SCD. We will report the complete results in the revised version.

For the concerns:

Concern 1: How to use LLMs for constructing the collaborative graph?

Thank you for your forward-thinking perspective. We believe that LLMs have significant potential in constructing educational collaboration graphs by designing agents that can identify collaborative learners based on educational prior knowledge and psychological assumptions. Existing studies [2,3,4] have demonstrated the use of LLMs to predict and analyze graphs in text, which can provide valuable insights for constructing collaboration graphs among learners in educational settings. We consider this a valuable research direction and plan to explore it in future studies.

Concern 2: Some typos.

Thank you for pointing out this issue. We will correct the symbols in the revised paper.

Concern 3: Confusion on baselines with duplicate names.

We apologize for the confusion. The first line of the DCD results does not include the tree structure, aligning with the settings for the ASSIST and Junyi datasets, which do not provide tree structures (as described in the "Settings" section). The second line of DCD results includes the tree structure. We will correct this in the revised version and label the second set as DCD_tree.

For the questions:

Question 1: Explain the difference of "collaboration" between education and recommendation.

The core idea in both contexts is to model similarities between entities to aid predictions. In the educational scenario, "collaboration" considers cognitive similarities between learners, aiming to diagnose the learner's cognitive state rather than predict the final outcome of answering questions. In a recommendation system, "collaboration" models the similarity of preferences among users, aiming to predict similar users' decisions on the same item through similarity.

Question 2: How to construct conceptual relations from ASSIST and NIPS2020EC datasets for running RCD?

The two datasets do not provide the concept maps required by RCD. Instead, we constructed the conceptual relations from exercise data using the statistical tool (https://github.com/bigdata-ustc/RCD) proposed by RCD.

Question 3: Details about the case study.

The example questions in the case study are randomly selected without a specific recommendation strategy, aiming to showcase the model's performance and interpretability in cold-start scenarios. Regarding "comparable situations", note that during inference, all model parameters remain fixed, meaning learners do not gain new knowledge. We will provide more details in the revised version.

[1] Towards the Identifiability and Explainability for Personalized Learner Modeling: An Inductive Paradigm[C]//Proceedings of the ACM on Web Conference 2024. 2024: 3420-3431.

[2] Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Education[J]. arXiv preprint arXiv:2402.14293, 2024.

[3] Large Language Model (LLM)-enabled Graphs in Dynamic Networking[J]. arXiv preprint arXiv:2407.20840, 2024.

[4] Canonicalize: An LLM-based Framework for Knowledge Graph Construction[J]. arXiv preprint arXiv:2404.03868, 2024.

Thank you for your prompt feedback and for recognizing the value of our work!

We are pleased to have addressed your concerns and will revise the paper according to your suggestions. If you have any further questions, please feel free to bring them up for discussion. We will address your concerns as quickly as possible!

Once again, thank you for your support and constructive feedback.

Thank you for the detailed and constructive feedback! We answer your comments and questions below. Please let us know if you have additional questions.

Weakness 1: Robustness under data sparsity or noise scenarios.

Thank you for highlighting the robustness of CD in data sparsity or noise scenarios.

In fact, we have presented experiments under data sparsity and cold start scenarios in Fig3(a) and (b), which demonstrate that Coral exhibits better robustness compared to the baseline models.

Additional experiments with noisy data:
Due to the lack of noisy labels, current CD models often struggle with data noise issue. Educational data noise is typically caused by implicit learner errors or guesses, which are difficult to quantify. Therefore, existing methods often treat noise as latent trainable parameters (e.g., [1][2]), but they fail to validate the model's ability to handle noise since the extent of noise in the dataset is unknown. To address your concerns about noisy data, we explicitly simulate noisy training scenarios by randomly flipping the performance labels of 5% of learners in each training dataset (0 -> 1, 1 -> 0). The test data remains unchanged and is considered noise-free, allowing us to assess whether models trained on noisy data maintain robustness on a noise-free dataset. Below are some of the results:

These results clearly show that Coral exhibits greater robustness compared to baselines in noisy scenarios. We will add all experimental results and discuss the impact of noisy data in the limitations section of the revised paper.

Weakness 2: Limited computational efficiency.

Thank you for your feedback on computational efficiency.

Our paper has presented three effective solutions to improve the efficiency of Coral in Appendix F.2. These strategies significantly enhance Coral's computational efficiency. While they may not guarantee theoretical optimality, the experimental results demonstrate substantial improvements, which prove Coral can be used to some large-scale scenarios. For real-time response, co-disentangled cognitive state (line 249) can be precomputed and stored, with only retrieval needed during online computation. We will continue to optimize the model efficiency in future studies.

Weakness 3: Add three related baselines.

Thank you for pointing out this issue and giving us the opportunity to supplement baselines.

We conduct experiments for the three baselines you suggested on three datasets. The experimental setups, results and analysis are available in the Public Response (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66). Experimental results demonstrate that Coral consistently outperforms these baselines across all datasets. We will add the complete experimental results and introduce these significant related works in the revised version.

We hope these responses comprehensively address your points and enhance your understanding of our work. We greatly appreciate your constructive feedback and remain open to any further inquiries or suggestions for improvement. Thank you for your support and valuable contribution to improving our paper.

[1] Terry A Ackerman. Multidimensional item response theory models. Wiley StatsRef: Statistics Reference Online, 2014.

[2] https://ieeexplore.ieee.org/document/10027634

Thank you for your prompt feedback and for recognizing the value of our work!

We are pleased to have addressed your concerns and will revise the paper according to your suggestions. If you have any further questions, please feel free to bring them up for discussion. We will address your concerns as quickly as possible!

Once again, thank you for your support and constructive feedback.

Thank you for the positive remarks and insightful feedback! We address your comments and questions below. Please let us know if you have additional questions.

Weakness 1: Concerns on computational efficiency.

Thank you for acknowledging our discussion of the model's efficiency limitation.

In fact, we also provide alternative efficient solutions in Appendix F.2 to address this limitation, where we design three practical strategies to improve Coral's computational efficiency. Although these solutions may not guarantee theoretical optimality, the experimental results demonstrate their effectiveness.

Weakness 2: Disentanglement evaluations.

Thank you for your suggestion on evaluating disentanglement quality.

Our disentanglement involves decoupling each learner into $C$ vectors (as vector-level disentanglement) and using KL divergence at the element level to encourage independence between each dimension of each vector based on Eq.5 (as dimension-level disentanglement). The current paper assesses dimension-level disentanglement quality by: (1) quantitative correlations between dimension independence level (IL) and prediction performance (Fig.3(a)), (2) qualitative embedding visualizations (Fig.4(b)), and (3) an ablation study of dimension-level disentanglement by discarding the KL term ("w/o KL" in Tab.2).

To further address your concerns, we add two additional studies:

1. We set different values for $\beta$ to explore the effect of varying disentanglement strengths (KL term in Eq. 5) on IL (i.e., dimensions' independence level of disentangled vectors) and performance, inspired by [1]. The experimental results on ASSIST are as follows:

The results indicate that as the disentanglement strength increases, the IL metric gradually increases, demonstrating that each dimension of learner representations becomes more independent. The model's predictive performance initially improves and then slightly decreases after reaching a threshold ($\beta = 0.5$). This suggests that both too low and too high disentanglement strengths can affect the model's performance.

2. We conduct ablation studies to remove the vector-level disentanglement by representing each learner with a single $C$-dimensional vector instead of the original $C$ d-dimensional vectors, denoted as "w/o VecDis". The experimental results on NIPS2020EC are as follows: |Model|ACC|AUC|RMSE| |-|-|-|-| |w/o KL|0.71026|0.78990|0.44382| |w/o VecDis|0.71523|0.78331|0.43412| |Coral|0.71622|0.79103|0.43200|

These results demonstrate the effectiveness of vector-level disentanglement. We will report the full results in a revised paper.

Weakness 3: How to address potential negative transfer impact in the collaborative graph.

Our model addresses potential negative transfer through attention weights (see Eq.8 for $s_{u,v}$), as discussed on lines 230-233. When computing attention between learners $u$ and $v$, both cognitive similarity and contextual similarity are considered. This approach ensures that attention is low for non-similar and non-collaborative neighbors, thus mitigating negative transfer impacts.

We conduct additional ablation experiments to further support this clarity:

The model without attention (w/o attention) performs worse compared to Coral, demonstrating the effectiveness of attention in managing negative transfer. We will include the full results in the revised paper and emphasize this technical advantage.

Question 1: Concerns on hyper-parameter sensitivity experiments.

Thank you for your feedback.

Current paper already included hyper-parameter experimental results (Fig.3) and findings (lines 323-328) on the number of neighbors $K$.

Additionally, we conduct experiments on different embedding dimensions $d$, as detailed in the Public Response (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=4eDEmw2q38).

Question 2: Data sparsity impacts.

Current paper provides experimental results of Coral with 20%, 40%, 60%, and 80% sparsity (Fig.3(a)). To fully address your concerns, we further conduct experiments with 5% and 10% sparsity as follows:

The experiments show the robustness of Coral. For 0% sparsity, current CD research that rely on practice data cannot effectively handle this scenario due to the need for initial data from each learner.

Question 3: Interpretability.

Each learner is disentangled into $C$ vectors, so that the disentanglement representation can naturally correspond to a specific knowledge, e.g., $z_{u}^{(c)}$ is $u$'s cognitive state of knowledge $c$.

An additional interpretability case study is given in Appendix E.

Limitations:

Thank you for your feedback. We supplement with a PISA 2015 data to expand dataset diversity (see the Public Response). To address data privacy, federated learning or cross-domain modeling can be integrated to enhance Coral's privacy-preserving capabilities. For fairness, we will ensure equitable outputs by identifying and mitigating the impact of sensitive attributes on model predictions. In the future, we will also design more powerful disentanglement algorithms.

We will include a discussion on these technical limitations and ethical considerations.

[1] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.

Thank you for recognizing the value of our work! We are pleased to have addressed most of your concerns and will incorporate the suggested modifications into the revised version.

We apologize for the misunderstanding regarding Question 2. Your further explanation helped us better understand this concern. Regarding Question 2, we would like to first clarify that such 5% and 10% cold-start phenomena are rare in real-world platforms unless it is a completely new platform. Current CD research focuses on learner-side cold-start scenarios where:

1. The learning platform has already accumulated rich learner data, and only a portion of learners are new [1].
2. A new business domain lacks learner data, but there is existing learner data from other related businesses [2].

In fact, our paper's Figure 3(b) has already validated the model performance under the second cold-start setup, where the training data is the mature domain and the test scenario is the cold-start domain.

Based on your detailed feedback, we further conduct the suggested cold-start experiments: selecting the earliest 5% and 20% of exercise data from learners in chronological order to train the model and then validating performance.

The results remain consistent with the 5% and 10% sparsity settings. We infer that there are three main reasons for these experimental outcomes:

1. In online learning, most learners have very few records. Thus, low sparsity rates like 5% and 10% result in almost no data, with an average of only 3 and 6 records per learner in our ASSIST dataset. This sparsity situation closely resembles that of new learners with minimal data.
2. CD assumes a learner’s cognitive state remains constant, meaning the order in which questions are answered does not affect performance [3]. This means that for CD models, there is little difference between using the earliest 5% and 10% of data (cold-start) based on time and using a randomly selected 5% and 10% of data (data sparsity).
3. Online education allows learners to practice autonomously, so their early data typically does not suffer from focusing on only a small subset of questions. This enables us to train parameters for most questions even with a limited amount of data.

Thank you for your feedback and for the further explanation, which helped me improve our paper and better understand your concerns. We sincerely hope that our further clarification addresses your concerns.

[1] BETA-CD: A Bayesian meta-learned cognitive diagnosis framework for personalized learning. AAAI'23.

[2] Zero-1-to-3: Domain-Level Zero-Shot Cognitive Diagnosis via One Batch of Early-Bird Students towards Three Diagnostic Objectives. AAAI'24.

[3] Towards a New Generation of Cognitive Diagnosis. IJCAI'21.

Thank you for the positive remarks and insightful feedback! We answer your comments and questions below. Please let us know if you have additional questions.

Weakness 1: Concerns on data diversity.

Thank you for your concerns about the datasets.

We would like to emphasize that our datasets are diverse, covering both K-12 scenarios (ASSIST) and online learning scenarios (Junyi and NIPS2020EC). These datasets are also representative, as they are benchmark datasets in CD research. Nearly all top-tier work (e.g., [1][2][3]) uses these datasets for experiments.

Following your suggestion, we further add a new dataset for experiments. The experimental results are detailed in the Public Response (https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66), supporting our model's effectiveness across diverse domains.

Weakness 2: Add baselines.

Thank you for your feedback.

We add four related baselines and compare them across three datasets. The experiments settings, results and model comparison analysis are listed in Public Response （https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66), demonstrating the superiority of Coral over these baselines. We will add complete results and detailed model comparison analysis in a revised version.

Weakness 3 and Question 3: Concerns on hyper-parameter sensitivity experiments.

Thank you for your concern on hyper-parameter sensitivity experiment. We would like to emphasize that our paper already includes hyper-parameter experiments (Fig.3(c)) on the number of neighbors $K$.

Additionally, we conduct experiments on different embedding dimensions $d$, with values of 124, 200, 256, 521, and 1024. The experimental results of Coral on ACC are listed in Public Response（https://openreview.net/forum?id=JxlQ2pbyzS&noteId=XpCCzTJX66), demonstrating that within an appropriate range, the dimension $d$ does not significantly affect Coral's performance. We will include the complete results in the revised version.

Weakness 4 and Question 4: Interpretability concerns.

We have already provided an interpretable example in Appendix E. This example demonstrates how Coral reasonably infers each learner's future performance by referencing the cognitive states of similar learners, thereby providing educators with a basis for their inferences.

Additionally, we would like to clarify that Coral's interpretability comes from its ability to use cognitive states of similar learners to predict how learners will perform on unfamiliar problems. Disentanglement is a technique to enhance the modeling of collaborative relations, not the source of interpretability. We infer that you might be misled by "co-disentanglement" in line 77, which means aligning learner representations from two perspectives to optimize collaborative relationship modeling, thereby enhancing interpretability. We will make this insight clearer in a revised paper.

Weakness 5 and Question 5: Overfitting issue.

Thank you for the insightful feedback.

We use an early stopping strategy to mitigate overfitting, stopping training when model's ACC (validation metric) on validation set stabilizes. Each model is trained 5 times with a repartition of data, and the average score is reported, similar to 5-fold cross-validation. We provide Coral's ACC error bars for 5 runnings on ASSIST, Junyi, and NIPS2020EC, $\pm 0.00023$, $\pm 0.00017$, and $\pm 0.00031$, ensuing robustness.

In fact, our paper verified Coral's robustness in both sparse and diverse scenarios: Fig. 3(a) shows experiments on data sparsity, showing Coral's robustness in handling overfitting with insufficient data. Fig. 3(b) shows cold start experiments where new knowledge is introduced to the test data, increasing test scenario diversity, proving the model's robustness even when the training data lacks diversity.

Weakness 6 and Question 6: Ethical concerns.

Thank you for raising these important ethical concerns. To address data privacy, federated learning or cross-domain modeling can be integrated to enhance Coral's privacy-preserving capabilities. For fairness, we will ensure equitable outputs by identifying and mitigating the impact of sensitive attributes on model predictions. Additionally, LLM-based agents are expected to maintain learner autonomy and agency. We will include a discussion of these ethical considerations in the revised version of the paper.

Question 1: Dataset selection criteria.

The data selection criterias include:

1. Authenticity: We use 3 real-world, open-source datasets, as detailed in Appendix D. These datasets reflect real learner interactions.
2. Diversity: Our datasets span different educational domains. The ASSISTments dataset targets K-12 education, while the Junyi Academy and NIPS datasets focus on online learning environments with global participants. This ensures coverage of varied learning contexts.
3. Representativeness: These datasets are established benchmarks in CD, widely used in current research for consistent evaluation.

Obviously, our datasets adequately represent diversity of learning environments and learner behaviors.

Question 2: Baseline descriptions.

We selected baselines from three aspects:

1. Educational Psychology Methods: IRT and MIRT, foundational in cognitive diagnosis,diagnose learners' cognitive states using psychology priors.

2. Machine Learning Approaches: PMF, NCDM, KaNCD, and RCD represent key methods using machine learning techniques to model student-item interactions.

3. Variational or Disentangled Methods: DCD and ReliCD, which are related to our solution.

We will add detailed descriptions of each baseline in the Appendix of the revised version.

[1] Disentangling cognitive diagnosis with limited exercise labels. NeurIPS'24.

[2] Hiercdf: A bayesian network-based hierarchical cognitive diagnosis framework. SIGKDD'22.

[3] RCD: Relation map driven cognitive diagnosis for intelligent education systems. SIGIR'21.