DisenGCD: A Meta Multigraph-assisted Disentangled Graph Learning Framework for Cognitive Diagnosis

Response to Q.1

The related work on disentangling graph representation learning is as follows, and we will add this to the revised paper:

1. Learning potential representations of disentangling in complex graph networks to achieve model robustness and interpretability has been a hot topic in recent years. Researchers have put forward a lot of classic neural network disentanglement approaches (e.g., DisenGCN[R1], DisenHAN[R2], DGCF[R3], DCCF[R4], DcRec[R5], etc.) to address this challenge. Some representative approaches will be briefly introduced.

2. For example, in DisenHAN[R1], the authors utilized disentangled representation learning to account for the influence of each factor in an item. They achieved this by mapping the representation into different spatial dimensions and aggregating item information from various edge types within the graph neural network to extract features from different aspects； In DCCF[R4], the authors introduced global intent disentanglement into graph contrastive learning, extracting more fine-grained latent factors from self-supervised signals to enhance model robustness； DcRec[R5] disentangles the network into a user-item domain and a user-user social domain, generating two views through data augmentation, and ultimately obtaining a more robust representation via contrastive learning.

3. Despite many approaches suggested, they were primarily applied to bipartite graphs to learn different representations from different perspectives, for learning more comprehensive representations. While this paper aims to leverage disentanglement learning to mitigate the influence of the interaction noise in the interaction graph, and thus we proposed a meta multigraph-assisted disentangled graph cognitive diagnostic framework to learn three types of representations on three disentangled graphs. By doing so, the influence of the noise on exercise and concept learning can be well alleviated.

[R1]Jianxin Ma, et al., Disentangled Graph Convolutional Networks. 
[R2]Yifan Wang, et al., DisenHAN: Disentangled Heterogeneous Graph Attention Network for Recommendation. 
[R3]Xiang Wang, et al., Disentangled Graph Collaborative Filtering. 
[R4]Xubin Ren, et al., Disentangled Contrastive Collaborative Filtering.
[R5]Jiahao Wu, et al., Disentangled Contrastive Learning for Social Recommendation

Response to Q.2

We would like to answer your question from the following three aspects:

• In the Appendix, we indeed provided a brief introduction to the three datasets (ASSISTments, Math, SLP). However, this introduction may not be detailed, thus we will give more information about the datasets in the revised paper, including their collectors, the contained subjects, and many other attributes.
• The cognitive diagnosis task is to assess the true knowledge level of students. To achieve this, this task is generally solved by modeling the students' answer prediction task, which is a binary classification task. Therefore, we use the evaluation metrics of classification to measure the diagnosis accuracy.
• The utilized metrics include ACC, RMSE, and AUC, which are widely used for the classification tasks. The three metrics are also widely adopted in many previous CD approaches.

Response to Weak.3

As you suggested, we added three variants of the DisenGCD to show the effectiveness of the added graphs, which are as

• DisenGCD(w/o both) is same as DisenGCD but learns exercise and concept representations directly through native embedding modules(NEMs) .
• DisenGCD(w/o $\mathcal{G_R}$) is same as DisenGCD but its exercise representation is learned through a NEM.
• DisenGCD(w/o $\mathcal{G_D}$) is same as DisenGCD but its concept representation is learned through a NEM.

Table II in global.pdf summarizes the results of all variants and the DisenGCD on the ASSISTments dataset.

As can be seen, the DisenGCD outperforms both DisenGCD(w/o $\mathcal{G_D}$) and DisenGCD(w/o $\mathcal{G_R}$), which indicates the effectiveness of learning exercise/concept representations by the GAT on added graphs. Moreover, the performance leading of DisenGCD(w/o $\mathcal{G_D}$) and DisenGCD(w/o $\mathcal{G_R}$) over DisenGCD(w/o both) can further validate the effectiveness of the added graphs. In addition, the effectiveness of the DisenGCD can be indirectly validated through the comparisons of DisenGCD(w/o both) and DisenGCD(I), as well as DisenGCD(w/o $\mathcal{G_D}$) and DisenGCD(Is+Rec).

In summary, the above comparison validates the effectiveness of the added graphs and the proposed disentangling learning framework.

Response to Q.4

As you suggested, other three types ofGNNs (i.e., GCN, GraphSage, and HAN) are used as the graph model backbones to replace the GAT module in DisenGCD. The variants of DisenGCD, utilizing GCN, GraphSage, and HAN as the backbones, are denoted as DisenGCD(GCN), DisenGCD(GraphSage), and DisenGCD(HAN).

To observe their influence on the proposed approach, these three variants were executed on the Math dataset, whereas the other two datasets were not used due to the time limit. As a result, Table I:Upper in global.pdf summarizes the results of these variants and the DisenGCD. We can find that the variant DisenGCD(GraphSage) can obtain competitive performance than state-of-the-art approaches, while DisenGCD(GCN) and DisenGCD(HAN) cannot. Despite its competitive performance, DisenGCD(GraphSage) is still worse than DisenGCD, which indicates that the GAT is most suitable for the proposed framework.

In summary, the results show the GAT is a suitable and optimal choice among these four GNNs for the proposed DisenGCD, and it is reasonable and effective for the proposed DisenGCD to adopt the GAT to learn the representations.

Response to Q.5

Thanks for pointing out the topos. We will correct these typos and double-check the paper.

We appreciate your valuable feedback and will revise the paper based on the above.

Thank you for your positive feedback. We greately appreciate your recognition of our efforts to address your concerns.

Response to Q.1

As you suggested, we added KaNCD [R1] to be compared with the proposed DisenGCD, and the comparison results are summarized in Table I:Upper in global.pdf, where only the Math dataset is used because there is no enough time to validate on other two larger datasets.

As you can see, in addition to KaNCD, we have also compared our approach with SCD [R2] and KSCD [R3], which are other two recently published CDMs. As shown in Table I:Upper in global.pdf, the proposed DisenGCD still performs better than KaNCD and other two state-of-the-art CDMs, further validating the effectiveness of the proposed DisenGCD.

In the future revised paper, we will add these CDMs to the comparison on all three datasets.

[R1] F. Wang, Q. Liu, E. Chen, Z. Huang, Y. Yin, S. Wang, and Y. Su, “Neuralcd: a general framework for cognitive diagnosis,” IEEE Transactions on Knowledge and Data Engineering, 2022.
[R2] Wang S, Zeng Z, Yang X, et al. Self-supervised graph learning for long-tailed cognitive diagnosis[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(1): 110-118.
[R3] H. Ma, M. Li, L. Wu, H. Zhang, Y. Cao, X. Zhang, and X. Zhao, “Knowledge-sensed cognitive diagnosis for intelligent education platforms,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022, pp. 1451–1460.

Response to Q.2

To solve your concern, three GNNs (GCN, GraphSage, and HAN) are used to replace the GAT in the proposed DisenGCD, which are termed DisenGCD(GCN), DisenGCD(GraphSage), and DisenGCD(HAN), respectively. Then, we compared the three approaches with the proposed DisenGCD on the Math dataset, where the other two datasets were not used due to the time limit. The comparison of them is presented in Table I:Upper in global.pdf.

As shown in Table I:Upper, the GCN-based DisenGCD achieves the worst performance, followed by DisenGCD(HAN). The GraphSage-based DisenGCD holds a competitive performance to yet is still worse than the proposed DisenGCD (based on GAT).

The above results show the GAT is a suitable and optimal choice among these four GNNs for the proposed DisenGCD, but we think the GAT may be not the best choice because there exist other types of GNNs that can be integrated with DisenGCD, showing better performance.

In summary, it is reasonable and effective for the proposed DisenGCD to adopt the GAT to learn the representations.

Response to Q.3

The design of the learnable propagation paths is inspired by the success of meta graph. And the propagation paths are selected by the adopted routing strategy, which learns a value for each candidate propagation path. Afterward, these propagation paths that satisfy the threshold will be automatically kept. Therefore, the selection of propagation paths is achieved by the model learning.

As for the influence of the propagation path number on the proposed approach, it can be found in line 179 that the path number is determined by the number of hyper nodes (i.e., P). Therefore, this influence can be investigated by analyzing the influence of the number of hyper nodes (i.e., P).

In fact, we have investigated the influence of P on the proposed DisenGCD in Appendix B.4. As shown in Figures 6 and 7 in Appendix B.4, the proposed DisenGCD is not very sensitive to the number of hyper nodes, where DisenGCD obtains a promising performance when P is 5. That indirectly indicates that the proposed DisenGCD is not sensitive to the number of candidate propagation paths.

In addition, for more convincing, the influence of different P on the robustness of the proposed DisenGCD is also analyzed as shown in Fig.1(a)&(b) in global.pdf, where only the Math dataset is used due to the time limit. Fig.1(a)&(b) present the results of RCD and DisenGCD with different P obtained under different ratios of noise interactions (1%, 30%, and 50%). As can be seen, the robustness of DisenGCD is a bit sensitive to different P, and the DisenGCD achieves the most promising robustness when P is equal to 5. That indirectly indicates that the proposed DisenGCD is sensitive to the number of candidate propagation paths.

In summary, the performance of the proposed DisenGCD is not sensitive to the propagation path number while its robustness is sensitive. Nevertheless, the proposed DisenGCD can obtain promising performance and robustness when P is set to 5.

We appreciate your valuable feedback and will revise the paper based on the above.

Thank you for your quick feedback and for contributing to our improved score. We greatly appreciate your efforts. We apologize that some issues still need improvement, and we would like to resolve them as follows.

Response to Weakness 1

To investigate the computational efficiency of the proposed approach, we have compared it with RCD on ASSISTments and Math datasets in terms of model inference time and training time. Table III in global.pdf presents the overall comparison, where the time of the proposed DisenGCD under different hyperparameters P is also reported.

As can be seen, DisenGCD's inference time is better than RCD's. This indicates that the proposed DisenGCD is more efficient than RCD, further signifying that it is promising to extend DisenGCD to dynamic CD. It can be seen from Table III:Lower: although a larger P will make DisenGCD take more time to train the model, the proposed DisenGCD achieves the best performance on both two datasets when P=5 and its runtime does not increase too much, which is much better than RCD.

In summary, the above comparison shows the model efficiency superiority of the proposed DisenGCD.

Response to Weakness 2

We have to argue that the diagnostic function in the proposed DisenGCD is interpretable.

Firstly, $\mathbf{h}\_{si} = F_{si}(\overline{\mathbf{S}_i}+ \overline{\mathbf{C}_k})$ denotes the combination of the learned student representation and the learned concept representations, which is similar to the combination in RCD, i.e., $Concat(\mathbf{S}\_i, \mathbf{C}_k)$. As a result, $\mathbf{h}\_{si}$ can be seen as the student's mastery of each knowledge concept.

Secondly $\mathbf{h}\_{ej} = F\_{ej}(\overline{\mathbf{E}_j}+\overline{\mathbf{C}_k})$ denotes the combination of the learned exercise representation and the learned concept representations, which is also similar to the combination in RCD for exercises, i.e., $Concat(\mathbf{E}\_j, \mathbf{C}_k)$. Therefore, $\mathbf{h}\_{ej}$ can represent the exercise difficulty of each concept.

Thirdly, $\mathbf{h}\_{simi} = \sigma(F\_{simi}(\mathbf{h}\_{si}\cdot\mathbf{h}\_{ej} ))$ is mainly used to measure the similarity between $\mathbf{h}\_{si}$ and $\mathbf{h}\_{ej}$ via a dot-product followed by an FC layer and a Sigmoid function. Therefore, a higher value in each bit of $\mathbf{h}\_{simi}$ represents a higher mastery on each concept, further indicating a higher probability of answering the related exercises.

Finally, $\hat{r\_{ij}} = (\sum Q_k\cdot \mathbf{h}_{simi})/\sum Q_k$ has a similar idea to NCD to compute the overall mastery averaged over all concepts contained in exercise $e_j$.

In summary, the proposed model is well-interpretable, as agreed by Reviewer 2m1M.

Response to Weakness 3

We have to admit that data privacy and security are indeed a crucial problem in intelligent education.

However, existing cognitive diagnosis models did not explore this. To the best of our knowledge, we think the following two potential techniques can be applied to CD to solve data privacy and security issues.

1. The first is to apply the differential privacy technique. Differential privacy [R1] protects individual data by adding noise, ensuring personal information can’t be easily inferred from data analysis. Differential privacy is widely used in fields like computer vision [R2], recommender systems [R3,R4], and many others [R5].

Among these, the recommendation task is most related to cognitive diagnosis (CD), and thus the differential privacy can be easily extended to CD by summarizing the experiences of these approaches. A successful example of extending techniques in the recommendation to CD is the MF [R6] model.

2. The second is to apply the Federated Learning technique. It trains a model across multiple devices without sharing data, improving privacy and efficiency. It has succeeded in many domains, including recommender systems [R7].

According to its successful experiences in the recommendation, federated learning can also be extended to CD. Moreover, a few studies [R8] have applied federated learning to knowledge tracing (KT), making good progress. KT is the most related task to CD in intelligent education; thus, their experiences can be brought to CD to solve data privacy and security problems.

To sum up, the proposed DisenGCD and existing CDMs could be combined with differential privacy and federated learning to solve the data privacy and security problem in intelligent education. We will discuss this problem in revised paper.

[R1] Dwork, Differential privacy. 
[R2] Mixed differential privacy in computer vision. 
[R3] A differential privacy framework for matrix factorization recommender systems. 
[R4] Applying differential privacy to matrix factorization. 
[R5] Differential privacy: A survey of results.
[R6] Matrix factorization techniques for recommender systems.
[R7] Federated recommendation systems.
[R8] Federated deep knowledge tracing.

Response to Q.1

We must admit that the design of the proposed DisenGCD did not consider the scenery of dynamic changes in knowledge structures. However, the proposed DisenGCD and RCD are trained in an inductive manner in graph learning. Therefore, it is feasible for DisenGCD and RCD to handle the scenery of introducing new concepts, but their performance is unknown.

To investigate this, we conducted the experiments on the Math dataset due to the time limit. Specifically, interactions and exercises related to 80% of knowledge concepts were not masked during the training, and we tested the model on the remaining interactions and exercises. The comparison results of the DisenGCD and RCD are presented in Table I:Lower in global.pdf.

We can see DisenGCD exhibits poor performance on such this scenery, which is poorer than RCD. That indicates that the proposed DisenGCD cannot adapt to dynamic changes in the knowledge structure.

The reason may be that the propagation paths learned in the meta-multigraph are not suitable for newly added knowledge concepts. Since adding 20% of concepts changes the topology of the interaction graph, while different-topology graphs generally need to learn different propagation paths. This fact can be found in Figure 4(a) in the submitted paper, stated in [R1].

In short, the proposed DisenGCD may not adapt to the scenery of introducing new concepts well due to the utilized learnable meta multigraph. In future work, we would like to explore how to address this issue to handle dynamic changes in knowledge structures.

[R1] Chao Li, Hao Xu, and Kun He. Differentiable meta multigraph search with partial message propagation on heterogeneous information networks. 

Response to Q.2

Actually, setting P to 5 is by trial and error on the DisenGCD. In Appendix B.4, we have discussed the influence of different P on the performance of DisenGCD. Figures 6 and 7 plot the results of DisenGCD obtained under different P (4, 5, 6, and 7) on the two datasets. It can be found that DisenGCD can achieve the optimal performance on both two datasets when setting P to 5.

However, we did not explore its influence on the computational efficiency. The most intuitive is that a larger P will cause worse computational efficiency.

To investigate this, Table III:Lower in global.pdf presents the performance and runtimes (for training) of RCD and DisenGCD under different P. As can be seen, it is consistent with the above assumption that a larger P will make DisenGCD take more time to train the model. The proposed DisenGCD achieves the best performance on both two datasets when P=5, and the runtime increase from P=4 to P=5 is not too much and is acceptable. Besides, Fig.1(a)&(b) in global.pdf show that the proposed DisenGCD can obtain the most promising robustness when P=5.

Therefore, in this paper, we set P to 5 for a comprehensive consideration of model performance, model robustness, and model efficiency.

Response to Q.3

To figure out the robustness of the proposed DisenGCD to exercise-concept noise, we conducted experiments on the Math dataset, where 1%, 10%, 20%, and 30% of exercise-concept relations are randomly removed as or added as noise relations, respectively. (Note that larger dataset) Fig. 1(c)&(d) in global.pdf compares the performance of RCD and DisenGCD on Math under different ratios of exercise-concept noise.

As can be seen, it is unexpected that the proposed DisenGCD exhibits better robustness than RCD to the added exercise-concept noises. However, different from against interaction noise (as shown in Figure 3(a) in the submitted manuscript), the robustness superiority/leading of DisenGCD over RCD does not increase as the ratio of noise increases, where the leading performance becomes smaller when a larger portion of noises are added.

It is reasonable because the added exercise-concept noise affects the learning on two graphs (the interaction graph and the relation graph), which further affects the learning of student and exercise representations. Since the learning of the concept representation is not influenced, DisenGCD still outperforms RCD. However, when more noise is added, DisenGCD's robustness against exercise-concept noise will not be better than that of DisenGCD against interaction noise, where the learning of two graphs is affected.

Response to Q.4

As for the first sub-question, to be honest, we have not investigated the performance of the proposed approach in scenarios where complex, cross-entity relationships. To answer your concern, we have tried our best to check many educational datasets， but we indeed cannot find a suitable dataset that holds cross-entity relations. Therefore, we are sorry that we cannot answer your question through experiments. However, as far as we are concerned, the DisenGCD can extend to handle cross-entity relations by disentangling more graphs. That may make promising performance, but its complexity and efficiency will be criticized. We also would like to answer your question if you can recommend a specific dataset.

As for the second, we would like to state that the compared RCD approach is essentially a unified graph approach, where all representations are learned and exchanged in an implicit graph. As shown in the submitted manuscript, the proposed DisenGCD exhibits better performance than the RCD. In addition, we have also compared DisenGCD with another unified graph-based approach, i.e., SCD [R1], on the Math dataset, where the results are presented in Table I:Upper in global.pdf. We can find that the DisenGCD still outperforms SCD. In summary, there may exist other novel unified graph approaches outperforming DisenGCD, but existing unified graph-based CDMs (RCD and SCD) cannot.

[R1] Wang S, Zeng Z, Yang X, et al. Self-supervised graph learning for long-tailed cognitive diagnosis[C].

Response to Weak.1 & Q.1

We admit that the proposed DisenGCD seems complex and needs more resources for its implementation because more graphs need to be handled. In future work, we would like to design a more efficient paradigm to learn robust representation in a unified graph.

As for Question 1, to be honest, we have to admit that the proposed DisenGCD is not designed for real-time dynamic cognitive diagnosis. However, as shown in some research [R1,R2], the graph-based CD approaches could be easily extended to knowledge tracing (KT, a task similar to dynamic cognitive diagnosis): for example, RKT [R1] is actually a variant of RCD in knowledge tracing, where the representations learned by graph NNs are attached as static exercise features used for sequence prediction.

Due to the time limit and our limited coding capabilities， we are sorry that we failed to extend our approach to dynamic cognitive diagnosis.

However, to extend graph CD approaches to KT or dynamic CD, a crucial problem to be solved is the graph-based model's efficiency. Therefore, to solve your concern indirectly, we make the statistics about the proposed DisenGCD's efficiency regarding its inference time. To this end, the inference time (seconds) of DisenGCD and RCD are compared in Table III:Upper in global.pdf.

As can be seen, the inference time of DisenGCD is better than RCD. That indicates that the proposed DisenGCD is more efficient than RCD, further signifying that it is promising to extend DisenGCD to dynamic CD.

[R1] Pandey, S., & Srivastava, J. (2020, October). RKT: relation-aware self-attention for knowledge tracing. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 1205-1214).
[R2] Yang, Y., Shen, J., Qu, Y., Liu, Y., Wang, K., Zhu, Y., ... & Yu, Y. (2021). GIKT: a graph-based interaction model for knowledge tracing. In Machine learning and knowledge discovery in databases.

Response to Q.2

The most important hyperparameter is the number of hyper nodes (i.e., P) in the meta-multigraph. To solve your concern, we would like to explore its influence from two aspects (i.e., on performance and robustness):

On the one hand, its influence on model performance has been discussed in Appendix B.4. As shown in Figures 6 and 7 in Appendix B.4, the proposed DisenGCD can achieve promising performance when the number of hyper nodes is 5, indicating setting P to 5 is reasonable.

On the other hand, we also analyzed the influence of different P (set to 4, 5, 6, and 7 ) on the model robustness under the Math dataset (ASSISTments is not used due to the time limit). Fig.1(a)&(b) in global.pdf shows the results of RCD and DisenGCD with different P obtained under different ratios of noise interactions (1%, 30%, and 50%). As you can see, when the number of hyper nodes is equal to 5, the robustness of DisenGCD is the most promising, whose performance under different ratios of noises is balanced better than RCD.

In summary, it can be seen that the proposed DisenGCD can achieve good performance and promising robustness when setting the number of hyper nodes to 5.

Response to Q.3

Other NNs can be integrated with the proposed framework yet the performance of them are not sure.

To validate this, we used three NNs (GCN, GraphSage, and HAN) to replace the GAT in the proposed approach, respectively, which are denoted as DisenGCD(GCN), DisenGCD(GraphSage), and DisenGCD(HAN). Due to the time limit, we only compared them with the proposed approach on the Math dataset, and the comparison results are summarized in Table I:Upper in global.pdf. It can be seen that the GAT enables the proposed framework to hold the best performance than other three variants, where DisenGCD(GraphSage) performs better than the other two.

The above comparison indicates that the proposed DisenGCD could be integrated with other neural network architectures and their performance may be improved when suitable NNs are adopted.

Response to Weak.3

According to your suggestions, we have added more recent CDMs, like SCD [R1], KaNCD [R2], and KSCD [R3] to the comparison. The comparison between them and DisenGCD is presented in Table I:Upper in global.pdf, where only the Math dataset is used due to the time limit.

As can be seen from Table I:Upper, compared to these recent CDMs, the proposed DisenGCD still exhibits better performance.

We will add these CDMs to the comparison in the revised paper.

[R1] Wang S, Zeng Z, Yang X, et al. Self-supervised graph learning for long-tailed cognitive diagnosis[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(1): 110-118.
[R2] F. Wang, Q. Liu, E. Chen, Z. Huang, Y. Yin, S. Wang, and Y. Su, “Neuralcd: a general framework for cognitive diagnosis,” IEEE Transactions on Knowledge and Data Engineering, 2022.
[R3] H. Ma, M. Li, L. Wu, H. Zhang, Y. Cao, X. Zhang, and X. Zhao, “Knowledge-sensed cognitive diagnosis for intelligent education platforms,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022, pp. 1451–1460.

We appreciate your valuable feedback and will revise the paper based on the above.