Towards Accurate and Fair Cognitive Diagnosis via Monotonic Data Augmentation

Q1: Validation on Diverse Datasets: Have you considered validating your model on more diverse and larger datasets to enhance the robustness and generalizability of your findings?

A1: Thank you for your suggestion. Firstly, we would like to clarify that the two datasets used in this paper are widely utilized in the field of cognitive diagnostics and are highly representative. They have been collected from real online learning scenarios and are extensively referenced in many papers, such as [1][2][3][4].

Furthermore, to address your concern, we have validated the effectiveness and robustness of our model on additional datasets. Specifically, we conducted new experiments on the ASSISTments2012 dataset, which was collected from ASSISTments, an online tutoring system in the United States between 2012 and 2013. The information regarding ASSISTments2012 is presented in Table 1:

Table 1. The statics of ASSISTments2012 dataset.

The results of CMCD on ASSISTments2012 are shown in Table 2 and Table 3. From these results, we observe that CMCD effectively enhances the accuracy and fairness of cognitive diagnostic models, demonstrating the effectiveness of our model.

Table 2. The accuracy result of CMCD and baselines (the higher, the better)

Table 3. The fairness result of CMCD and baselines (the lower, the better)

[1] Wang F, Liu Q, Chen E, et al. Neural cognitive diagnosis for intelligent education systems[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(04): 6153-6161.

[2] Wang F, Gao W, Liu Q, et al. A Survey of Models for Cognitive Diagnosis: New Developments and Future Directions[J]. arXiv preprint arXiv:2407.05458, 2024.

[3] Liu S, Shen J, Qian H, et al. Inductive Cognitive Diagnosis for Fast Student Learning in Web-Based Intelligent Education Systems[C]//Proceedings of the ACM on Web Conference 2024. 2024: 4260-4271.

[4] Yu X, Qin C, Shen D, et al. Rdgt: enhancing group cognitive diagnosis with relation-guided dual-side graph transformer[J]. IEEE Transactions on Knowledge and Data Engineering, 2024.

Q2: Practical Implementation: Can you provide more details on the practical challenges and solutions for implementing CMCD in real-world educational environments?

A2: Thank you for your feedback. In real-world educational settings, it is crucial to have access to users' response records, which may sometimes be restricted due to privacy considerations. It's important to note that for all cognitive diagnostic tasks currently, gathering users' response records is fundamental to research in the entire cognitive diagnostic field, the privacy issue is not within the scope of our current discussion in this paper. To address this privacy issue, we intend to integrate our approach with federated learning and differential privacy. This limitation has been discussed in Section 7, Conclusion and Discussion, of the paper.

Q3: Theoretical Guarantees: How do you plan to empirically demonstrate the theoretical guarantees of accuracy and convergence speed provided in the paper?

A3: Sorry for the confusion. In fact, besides theoretically validating the accuracy and convergence speed of our method, we have empirically verified these aspects as well.

• Regarding accuracy, we conducted experiments on different datasets and different cognitive diagnostic backbones (IRT, MIRT, NCDM), where we found that our method enhances the accuracy of the baseline on real datasets. These results are detailed in Section 6.2 RQ1. Furthermore, we performed ablation experiments (Section 6.2 RQ2) and discovered the effectiveness of the data augmentation strategy in improving accuracy within CMCD.

• Regarding convergence speed, we conducted experiments on real datasets. Specifically, we compared the convergence speeds (the epoch at which early stopping occurs) of different baselines on various backbones. The experiments revealed that we can achieve faster convergence significantly, as demonstrated in Section 6.2 RQ3.

Q4: Ethical Considerations: What measures have you considered to address data privacy and informed consent issues in future studies?

A4: Thank you for your thoughtful suggestion. We plan to integrate our method with the following three perspectives.

• Encrypted Communication: Ensure the use of secure encrypted channels during data transmission in the CD. This helps prevent data from being intercepted or tampered with during the transmission process, safeguarding the privacy and integrity of students' response records.

• Differential Privacy: By incorporating differential privacy techniques, we can introduce controlled noise to the data during the model training process. This aids in protecting the privacy of individual student data, preventing malicious users from inferring specific student information through the model's outputs.

• Federated Learning: By employing federated learning methods, we can distribute the model training process across multiple local devices rather than centralizing it on a single server. In federated learning, only encrypted parameters are shared during the model update phase, not the raw data. This approach ensures that student response records remain local and decentralized, safeguarding student privacy.

Q1: The relationship between the two proposed constraints for data augmentation

A1: Thank you for your explanation. The constraints of these two data augmentations reflect the two states in the monotonicity assumption: when a student answers a question correctly, their ability is assumed to be higher than when they answer the same question incorrectly (Hypothesis 1), and when a student answers a question incorrectly, their ability is assumed to be lower than when they answer the same question correctly (Hypothesis 2). These two constraints complement each other and work synergistically to better address the issue of data sparsity, thereby achieving fairer and more accurate cognitive diagnostics. To validate this point, we conducted corresponding experiments. Specifically, we performed ablation experiments on the Math dataset where we systematically removed the corresponding hypothesis strategies. The results are presented in Table 1 and Table 2.

From the tables, we observe that both constraints corresponding to the hypotheses can enhance both fairness and accuracy. This indicates that both constraints, which align with the monotonicity assumption, can facilitate more accurate and fair cognitive diagnostics. Importantly, when both constraints are applied (i.e., CMCD), better results are achieved compared to applying only one constraint. This suggests a synergistic effect between the two constraints, where they enhance each other's performance.

Table 1. The accuracy result of CMCD on IRT (the higher, the better)

Table 2. The fairness result of CMCD on IRT (the lower, the better)

Q2: How do these methods differ from data augmentation techniques used in other user modeling tasks?

A2: Sorry for the confusion. The differentiation of our CMCD approach from standard data augmentation techniques is primarily manifested in two key aspects:

• Monotonicity Assumption: The Monotonicity Assumption stands as a fundamental theoretical cornerstone in the realm of Cognitive Diagnostics (CD). It specifically asserts that a student's proficiency demonstrates a monotonic correlation with the likelihood of providing a correct response to an exercise. Building upon this premise, we have introduced two data augmentation constraints, with experimental results validating the efficacy of our approach.
• Theoretical Guarantees: We have provided theoretical assurances regarding the accuracy and convergence of the proposed data augmentation strategies. Detailed information on these theoretical guarantees can be found in section 5.2 of our work.

In terms of practical application, we have also adapted some traditional data augmentation methods to the domain of cognitive diagnostics (specific baseline details can be found in A.3). Comparative analyses between these traditional methods and our CMCD approach have revealed superior performance of CMCD in both accuracy and fairness. Furthermore, our method demonstrates accelerated convergence rates (specific details can be found in section 6.2, RQ1, RQ2). We intend to emphasize these pivotal points in the revision.

Q3: Is it possible to extend these methods to other user modeling tasks?

A3: Thank you for your thoughtful question. The core contribution of our paper is reflected in two aspects. Firstly, we conducted an analysis from a data perspective and identified that data sparsity could lead to unfairness and inaccuracy. We proposed achieving more accurate and fair cognitive diagnostics through the lens of data augmentation. Secondly, based on the unique monotonicity assumption in the field of cognitive diagnostics, we introduced two data augmentation constraints and provided theoretical guarantees.

Regarding the first point, drawing from our past experiences, we observed that the data distributions in other user modeling tasks, such as recommendation systems, and the field of cognitive diagnostics generally exhibit similar trends. This suggests that the approach of addressing data sparsity to enhance accuracy and fairness in user modeling can be transferred. However, in tackling the challenge of data sparsity, our paper uniquely utilized the cognitive diagnostics domain's distinctive monotonicity assumption to augment data. By integrating this with the cognitive diagnostics paradigm, we provided corresponding theoretical guarantees. This method cannot be directly extended. In the future, we will explore how to leverage the specific characteristics of user modeling tasks to effectively address data sparsity issues.

W1: I noticed that the experimental results at the end show an improvement in both the fairness and accuracy of diagnostics simultaneously, which seems to contradict the prevailing trade-off between fairness and accuracy. It would be beneficial for the paper to delve deeper into this issue, exploring whether this approach can be extended to other domains such as the field of recommender systems.

Thank you for your constructive feedback. In fact, in the realm of fairness research, there does exist a trade-off between fairness and accuracy. This implies that while traditional fairness-aware methods can enhance fairness, they also tend to reduce utility. We have validated this phenomenon through experiments. Specifically, Table 1 displays the accuracy results of CMCD and the baselines, while Table 2 showcases the fairness results. Among these baselines, CD+Reg, CD+EO, and CD+DP are all considered fairness baselines. We observe that these baselines have to some extent improved the fairness of the original cognitive diagnostic model but have simultaneously decreased its performance.

What sets our work apart in this article is our departure from traditional fairness approaches. We have approached the issue from a data perspective and, through extensive data analysis on real datasets, discovered that data sparsity can lead to inaccurate and unfair outcomes. The specific data analysis is presented in Section 4. This finding suggests that mitigating data sparsity in cognitive diagnostic tasks can alleviate the trade-off between fairness and utility. However, it is worth noting that in terms of the improvement in fairness, CMCD does not consistently outperform traditional fairness baselines and achieve state-of-the-art results. This discrepancy is due to our work considering both accuracy and fairness, highlighting the inherent trade-off between fairness and accuracy.

Finally, we discuss the potential extension of our work to other domains, such as recommender systems. The data distribution in cognitive diagnostic fields and recommender systems exhibits similar trends. Therefore, alleviating data sparsity issues can potentially enhance both fairness and accuracy simultaneously, making the data-driven approach transferrable. However, our method relies on the unique monotonicity assumption in cognitive diagnostic fields to enhance data, combined with the cognitive diagnostic paradigm to provide corresponding theoretical guarantees. This approach cannot be directly extended. Thank you once again for your feedback. We will incorporate the relevant discussion in the revision.

Table 1. The accuracy result of CMCD and baselines on NCDM （for the RMSE, MAE metric, the lower the better, for the AUC, ACC metric, the higher the better）

Table 2. The fairness result of CMCD and baselines on NCDM (the lower, the better)

W2: In the experimental section, I observed that the utility and fairness metrics are aligned. While I understand that fairness analysis is based on utility outcomes, this alignment could potentially mislead readers to some extent. Clarity on this relationship would enhance the understanding for the readers.

Sorry for the confusion. In this paper, the calculation of fairness metrics is based on user-oriented group fairness [1], which asserts that a fair model should deliver an equal level of utility performance across different user groups. Therefore, we express the performance gaps between the two groups accordingly. To simplify, we directly use performance metrics as a representation. To avoid any confusion, we will change our metric names in the revision. For example, in the fairness table, we will replace RMSE, MAE, AUC, ACC with GF(RMSE), GF(MAE), GF(AUC), GF(ACC).

[1] Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2021. User-oriented fairness in recommendation. In Proceedings of the Web Conference 2021. 624–632.