Advancing Personalized Learning with Neural Collapse for Long-Tail Challenge

Q1: However, the text lacks detailed information on the underlying mathematical principles of the Textmodality Collapse regularization, and a deeper explanation would be beneficial.

A1: Thank you for the reviewer's insightful comment. In our paper, the goal of TC regularization is to promote a more balanced and structured text feature distribution, thereby addressing the long-tail distribution problem by leveraging the constraints from the information of different categories. In Section 4.3, we explain the mathematical principles behind TC regularization from the perspective of gradient optimization and its role in constraining the text feature space.

Q2: However, the TC loss in Equation 14 and the loss in Equation 13 should be further explained and clarified.

A2: We further clarify the distinction between the TC loss in Equation 14 and the loss in 13. The gradient in 13 consists of an "intra-class cohesion" term and an "inter-class repulsion" term. In class-imbalanced learning paradigms, the gradient signals for underrepresented minority classes become dominated by the repulsion term, which creates geometric configuration biases that can result in suboptimal feature space organization for rare categories. Equation (14) explicitly shows that the TC loss adjusts text representations to ensure maximal angular separation, mitigating the adverse effects of imbalanced gradient updates.

Q3: However, the authors should visualize the statistical analysis of these two real-world datasets.

A3: In the TMWPL dataset, there are seven classes: Recall, Formulate, Identify, Represent, Implement, Inferences, and Analyze, with sample sizes of 5045, 1283, 717, 711, 405, 235, and 216, respectively. On the other hand, the PMTD dataset consists of eight classes: R-FR, I-Q, R-SR, F-I, F-F, U, I-H, and R-RR, with sample sizes of 1014, 880, 737, 518, 510, 389, 104, and 62, respectively.

Q4: The authors should explain why these four specific values of $\tau$ were chosen for the experiments.

A4: Firstly, $\tau$ represents the imbalance ratio of the data, defined as the ratio of the number of samples in the least frequent category to the number of samples in the most frequent category. In our paper, 0.03 corresponds to the $\tau$ of our original data, while 0.25 is the maximum $\tau$ that our data can support for training without compromising the model's performance. Beyond this threshold, the data volume is insufficient for effective training.

Q5: The authors should explain the possible reasons behind this unexpected result.

A5: The reviewer raised an interesting question. In the ablation experiments, the performance improvement of the "w/o prompt" condition compared to the baseline method may stem from the model's ability to more effectively construct the text feature space in the absence of a prompt. This suggests that removing the prompt could allow the model to be more flexible in feature extraction, thereby enhancing performance. Although this phenomenon is intriguing, we believe further experiments are needed to validate its correctness, and we plan to explore this as part of future work.

Q6: More experimental designs for our method.

A6: We followed the reviewer's suggestion and considered additional baselines to compare the effectiveness of our method. The experimental results are shown below. Under the comparison with the same model parameters and even larger model parameters, our method still demonstrates robust performance on long-tail distribution datasets. This further confirms that the introduction of neural collapse can constrain the learning of the model's class space, thus avoiding biases caused by data with a large number of samples.

Q7: In Equation (5), the symbol $\alpha$ is not explained.

A7: $\alpha$ is a constant parameter for scaling in our paper.

Q1: However, the assertion that NCAL is model-agnostic would benefit from further clarification.

A1: NCAL is designed to learn features that conform to the same simplex equiangular tight frame (ETF) structure by introducing TC regularization to optimize text distribution. As a result, NCAL serves as a plug-and-play framework. To further demonstrate its versatility, we present experimental results in Table 2, showcasing NCAL’s performance when integrated with different LLM architectures, such as Qwen2.5 and Llama3.1.

Q2: More detailed examples of its impact on real-world class imbalance problems would provide further credibility to the claims.

A2: Thank you for your valuable suggestion. We visualized the class imbalance statistics of the personalized learning data collected from real-world scenarios used in this paper (see Q5). We also conducted experiments on other open-source datasets to evaluate NCAL’s performance under different class imbalance datasets. These results further demonstrate its effectiveness in practical applications (see Q3).

Q3: However, incorporating additional personalized learning baselines or datasets.

A3: As suggested by the reviewer, we add a comprehensive study comparing four LLM methods and three open-source datasets, including: IITJEE NEET AIIMS Students Questions Data; MathDial; DialogID. These results demonstrate the effectiveness of NCAL in comparison with more baselines under various long-tail personalized learning datasets. These results are shown below:

Q4: Would it be possible to include additional visualizations?

A4: In Figure 1, we present the visualization to demonstrate the effectiveness of our method in improving the class feature space. The arrows indicate the direction of class centers, while the points represent sample features, with colors corresponding to the classes. In Figures 1 (a) and (c), it is visually evident that the baseline suffers from severe class representation imbalance. In (b) and (d), our method more uniformly learns the information for each class.

Q5: However, more information should be provided, such as the number of samples in each category, to give a more comprehensive view of the data distribution.

A5: We thank the reviewer for pointing out this issue. We will include the statistical data in the final version. The data distribution is as follows, and it is clear that the number of samples in different categories exhibits a distinct long-tail distribution.

Q6: It is recommended to include experiments comparing these methods with the approach proposed in this paper.

A6: We agree with the reviewer that data augmentation is indeed an effective approach for addressing long-tail distributions. However, data augmentation requires strict control over the augmentation strategy, as well as the quality and diversity of the generated samples. In contrast, NCAL has broader applicability. It is important to note that we sincerely cannot provide a baseline comparison with the paper ``Towards Accurate and Fair Cognitive Diagnosis via Monotonic Data Augmentation,” as the augmentation method used in that work is based on ID-type data (e.g., 0, 1, 2), which cannot be learned using LLM encoding.

Q1: However, the claim about mitigating class imbalance would benefit from further discussion or visualizations to highlight the improvement in class distribution. More visual evidence could strengthen the claim of enhanced category representation.

A1: We appreciate the reviewer’s comments. Below, we present the statistical distribution of the data used in our paper. Please note that these two datasets were collected from real-world scenarios, with PMTD containing eight categories and TMWPL comprising seven categories.

Q2: However, the mathematical formulation and the justification behind the introduction of TC regularization could benefit from further explanation, and how it mitigates the long-tail distribution issue.

A2: The goal of NCAL is to learn a simple ETF through the method of neural collapse, to construct a uniformly partitioned category feature space. To achieve this, we introduce TC regularization, which aims to constrain the feature distribution of category partitioning learned from text data across different categories, effectively addressing the long-tail distribution problem.

Q3: However, incorporating additional experiments to evaluate the method’s performance under imbalance rates and experimental details.

A3: On the TMWPL dataset, we conducted experiments based on the Qwen2.5-Math-Instruct architecture, evaluating the model's performance under different levels of long-tail distribution, ranging from low to high imbalance ratios (0.03, 0.08, 0.13, 0.18, 0.25). The corresponding performance results were 74.86, 75.01, 76.45, 78.06, and 76.86. The experimental results indicate that as the degree of long-tail distribution weakens, the model's performance improves.

Q4: Could you further explain Equation (10)? Are i and j texts from the same category or different categories?

A4: Yes, I will provide a more detailed explanation for Equation (10). The texts $i$ and $j$ come from different categories, and the goal is to constrain the feature representations of samples from different categories to approach a uniform distribution, i.e., $1/(N-1)$, where $N$ represents the number of categories.

Q5: Although the authors have validated the effectiveness of their method on two representative data-driven personalized learning datasets, it is recommended to conduct experiments on additional open-source datasets to more comprehensively assess the method’s generalization ability and robustness.

A5: As recommended by the reviewer, we have included a comprehensive study comparing three open-source datasets: IITJEE NEET AIIMS Students Questions Data, MathDial, and DialogID. The results, presented below, highlight the effectiveness of NCAL compared to additional baselines. The experimental results indicate that our method consistently outperforms others across all datasets.

Q6: The impact of different types of prompts on performance?

A6: Thank you for the reviewer’s suggestion. Based on the prompts considered in our manuscript, we have referred to related work and explored additional prompt forms, including base prompt (see Table 5 in our manuscript), w/o prompt, few-shot (introducing a few reference samples in the prompt), role play (personalized scene prompts), and only class name. The experimental results are shown below. The results demonstrate that our method maintains strong robustness and generalization across different prompt settings.

Q1: While the paper provides evidence for the performance improvements and generalization ability of NCAL, the claim of it being “model-agnostic” remains somewhat unclear.

A1: NCAL enhances feature learning by aligning with the simplex equiangular tight frame (ETF) structure through Text-modality Collapse (TC) regularization, which optimizes text distribution. This design ensures NCAL's compatibility with various models, making it a versatile plug-and-play framework. As evidenced in Table 2, we demonstrate its effectiveness across multiple LLM architectures, including Qwen2.5 and Llama3.1, further supporting its model-agnostic property.

Q2: More specific details on its application to different models and a deeper explanation of how it addresses class imbalance will help to reinforce these claims.

A2: We have further clarified the learning mechanism of NCAL. Specifically, NCAL aims to learn features that conform to the Equiangular Tight Frame (ETF) structure by introducing Text-Modality Collapse (TC) regularization. This optimization improves the distribution of text embeddings within the representation space of large language models (LLMs), thereby enhancing the robustness and generalization ability of LoRA fine-tuned LLMs under long-tail data distributions.

Q3: The authors should add ablation experiments to further discuss the impact of different imbalance ratios.

A3: As suggested by the reviewer, we conducted a comprehensive comparative study on different imbalance ratios. Specifically, we evaluated the model under imbalance ratios of 0.03, 0.05, 0.10, 0.20, and 0.25. The experimental results are shown below, indicate that as the imbalance ratio decreases from 0.03 to 0.25, the model's performance improves accordingly, increasing from 74.86 to 76.86.

Q4: Different forms of prompts can significantly impact the performance of LLM models. In Section 5.3, the authors examined the effect of using or not using prompts on the experimental results but did not further analyze the performance differences across different prompt formats.

A4: Following the reviewer's suggestion and related work, we validate our method with more prompt forms, including base prompt (see Table 5 in our manuscript), w/o prompt, few-shot (introducing a few reference samples in the prompt), role play (personalized scene prompts), and only class name（see Table 4）. The experimental results are shown below:

The experimental results show that different prompt forms have little impact on the model. The possible reason is that our method, using the TC regularization approach, can learn a good category feature space that is not overly influenced by the prompt during training and inference. This experiment further supports our motivation.

Q5: I recommend including more baseline methods, for data-driven personalized learning to further strengthen the comprehensiveness of the study.

A5: We thank the reviewer for pointing out this issue. We conducted extensive experiments based on the papers recommended by the reviewer. Unfortunately, we found that the codes for the methods in [1] and [3] have not been released, making it difficult for us to reproduce these methods within the short time frame for the rebuttal. Additionally, [2] is a survey paper, from which we selected the most recent representative state-of-the-art (SOTA) works for performance comparison. The experimental results are as follows, showing that our method still achieves SOTA performance compared to other SOTA baselines under the more challenging TMWPL dataset.

Q6: The concept of equidistant text representation (ETR) regularization needs further explanation. Does it share the same meaning as Textmodality Collapse (TC)?

A6: Thank you for the reviewer's comment. In our manuscript, the meaning of ETR is the same as TC, and we will correct this error in the final version.