Structural-Entropy-Based Sample Selection for Efficient and Effective Learning

We sincerely thank you for your appreciation of our idea and your insightful suggestions. We have carefully considered your feedback and revised the paper accordingly. Please see our response below.

Q1: Clarification of novelty and technical contribution.

A1: We have clarified the contribution of our work in the third and fifth paragraphs of the Introduction. The first contribution of our work is a node-level structural entropy metric that quantifies the importance of individual nodes in preserving the global structure. Existing sample selection methods focus primarily on local information and overlook global information, which can result in suboptimal selection. To address this limitation, we explored possible solutions and identified graph structural entropy as a metric that accurately captures the structure of a sample graph. However, it only provides a single value for the whole graph. It is challenging to decompose this metric to the level of individual nodes, limiting its utility for fine-grained, node-level selection. To overcome this, we used the Shapley value to decompose the structural entropy. However, this introduced another challenge: the exponential-time complexity of calculating the Shapley value. To address this, we reformulated the Shapley value and proposed a node-level structural entropy metric that enables linear-time calculation with respect to the edge number. Based on this, our second technical contribution is developing a structural-entropy-based sample selection method that tightly integrates both global and local metrics. The experiment results in Tables 1, 2, 3 of our paper clearly demonstrate that our method consistently improves state-of-the-art methods across all scenarios and tasks. We hope that our work can encourage a closer look at the incorporation of global information into sample selection and potentially spark interest in further exploration.

Q2: Justification for Eq. (4) and Eq. (5).

A2: We have fully revised the justification for Eq. (4) and added a theoretical justification and an ablation study to support the quantification of node-level structural entropy by Eq. (4). The theoretical justification is provided in Sec. 4.1 and Appendix C. We have proved that $S_e(u)$ provides a lower bound for the sample coverage. This result indicates that maximizing $S_e(u)$ during selection inherently improves sample coverage. Given the strong correlation between sample coverage and the empirical loss of learning algorithms [1], selecting samples with high node-level structural entropy effectively enhances model performance. The ablation study is provided in Appendix G, and the results on CIFAR10 are as follows. These results demonstrate that removing the second term slightly increases performance across all sampling rates. This finding supports our decision to remove the second term in quantifying node-level structural entropy.

For prioritizing high-difficulty samples (local information), previous research [2, 3] has demonstrated that training difficulty is an effective local metric for quantifying a sample’s impact on model performance. This is because prioritizing high-difficulty samples enables the model to focus on challenging cases, which are typically more informative for improving decision boundaries and generalization [2, 3]. We make this clearer in Sec. 4.2.

Q3: Combination methods of global and local metrics.

A3: We explore possible alternative methods to combine the global metric $S_e(u)$ and the local metric $S_t(u)$ other than the proposed multiplication. Specifically, we experiment with the sum ($S_e(u)+S_t(u)$), the harmonic mean ($\frac{S_e(u) S_t(u)}{S_e(u)+S_t(u)}$), and the maximum ($\max(S_e(u),S_t(u))$). The ablation study is provided in Appendix G, and the results on CIFAR10 are as follows. These results demonstrate that multiplication achieves the best average performance, validating our choice for combining the two metrics.

Q4: Detailed experimental setup.

A4: Thank you for pointing this out. We have clarified in Sec. 5.2 that the key difference between active learning and supervised learning lies in the absence of labels during sample selection, which makes the selection process more challenging. Additionally, we have revised the whole Sec. 5 and included more details on the experimental setup.

References

[1] Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-Centric Coreset Selection for High Pruning Rates. In Proceedings of the International Conference on Learning Representations, 2022.

[2] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep Learning on a Data Diet: Finding Important Examples Early in Training. In Proceedings of Advances in Neural Information Processing Systems, pp. 20596–20607, 2021.

[3] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning. In Proceedings of Advances in Neural Information Processing Systems, pp. 19523–19536, 2022.

We hope that our responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript.

We sincerely thank you again for your recognition of our work. As we mentioned in the previous response, we have revised the paper per your suggestions. We hope that these responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript.

If you have any additional comments, we will do our best to address them.

We sincerely thank you for recognizing the novelty of our paper and providing constructive suggestions. We have carefully revised our paper accordingly. Please see our response below.

Q1: Hypergraph experiments.

A1: We have conducted an experiment using hypergraphs to capture the relationships between samples. Following the previous work [1], we construct a hypergraph by adding a hyperedge for each node that contains the node itself and its $k$-nearest neighbors. Due to the lack of the structural entropy theory for hypergraphs, we apply clique expansion [2] to convert the hypergraph into a graph and perform our selection method on the resulting graph. The results on CIFAR10 are as follows. This hypergraph-based method slightly degrades performance compared to the graph-based method, but still performs better than the baseline methods in low-sampling-rate settings. We attribute this slight decline in performance to the information loss during clique expansion, for which a straightforward alternative is not currently available. We see significant potential for future work in developing structural entropy theories for hypergraphs and then applying our method to sample selection. We have included this detailed ablation study in Appendix G for further reference.

Q2: Definition of informative and representative samples.

A2: Thank you for pointing this out. We have now clarified in the Introduction that informative samples are those that significantly reduce model uncertainty and are crucial for improving the accuracy and robustness of the training process, while representative samples are those that preserve the diversity and overall distribution of the dataset [3].

Q3: Hyper-class representation.

A3: Thank you for suggesting using a hyper-class representation to better capture information at the sample level. We discuss this method in Sec. 6. While we believe it has significant potential and can inspire more follow-up work, we have not included it in our current study for two reasons. First, it is orthogonal to our main contribution in incorporating global information to select high-quality samples. Second, it is not straightforward to combine the hyper-class representation with our method because it is currently designed for tabular data. We have discussed this with the second author of the referenced paper (the primary implementer of the method). In the discussion, he indicated that adapting the hyper-class representation requires exploring how to handle image and text embeddings rather than tabular data used in the original paper, and fully exploring this idea definitely deserves a separate publication. We believe this is a highly promising research direction to explore in the future.

Q4: Lack of variance in experimental results.

A4: Thank you for the suggestion. We have added the variance of all experimental results in Appendix F. Due to the space limitations, here we only provide the results on CIFAR10.

Q5: Effectiveness and appropriateness of the comparison algorithms.

A5: Random selection is a robust method that requires no prior assumptions about what constitutes an ''optimal" sample, and it has consistently served as a strong baseline across various settings in prior evaluations [4, 5]. In contrast, more complex selection methods perform well in specific settings. For example, the methods that select the most difficult samples perform better when a large number of samples are selected, as they can enhance the model’s learning by exposing it to challenging samples. However, when the sample size is limited, these methods often perform worse because only focusing on difficult samples may lead to an unrepresentative subset that lacks coverage, which harms model generalization. To ensure a comprehensive and fair comparison, we mainly followed $\mathbb{D}^2$ pruning [5] to choose the comparison algorithms with diverse characteristics. We set the algorithm hyperparameters according to the configurations reported in the original papers.

Q6: Comparison between message passing (MP) and blue noise sampling (BNS).

A6: The hyperparameter tuning for MP requires a time-consuming grid search based on model training results. In our experiments, the optimal hyperparameter ranged from 0.1 to 100, and the grid search took $15$ times longer than our method (a single model training). We have now clarified the considerable time cost of MP in Sec. 5.4. Therefore, we use BNS in our method because it requires no hyperparameter tuning and still achieves performance comparable to that of MP when the sampling rate is greater than or equal to 20% (the differences range from -0.07% to -0.23%).

Q7: Explanation of Figure 4.

A7: Thank you for pointing this out. We make this clearer in Sec. 5.5 that the colored contours in Figure 4 indicate the high-density regions for each class. The overlapping regions contain ambiguous samples from different classes, which are important for the model to learn to differentiate. In Figure 4(b), $\mathbb{D}^2$ pruning misses important samples in these regions, limiting the model’s ability to differentiate samples in these regions. Conversely, in Figure 4(c), our method well covers the data distribution, leading to improved model training outcomes.

References

[1] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph Neural Networks. In Proceedings of the AAAI conference on artificial intelligence, pp. 3558-3565, 2019.

[2] Sameer Agarwal, Kristin Branson, and Serge Belongie. Higher Order Learning with Graphs. In Proceedings of the International Conference on Machine Learning, pp. 17-24, 2006.

[3] Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. Active learning by querying informative and representative examples. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(10): 1936–1949, 2014.

[4] Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A Comprehensive Library for Coreset Selection in Deep Learning. In Proceedings of International Conference on Database and Expert Systems Applications, pp. 181–195, 2022.

[5] Adyasha Maharana, Prateek Yadav, and Mohit Bansal. $\mathbb{D}^2$ Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning. In Proceedings of the International Conference on Learning Representations, 2024.

We hope that our responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript.

We sincerely thank you again for your recognition of our work. As we mentioned in the previous response, we have revised the paper per your suggestions. We hope that these responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript.

If you have any additional comments, we will do our best to address them.

We sincerely appreciate your valuable and helpful suggestions. Per your suggestions, we have carefully addressed the identified weaknesses. Please see our response below.

Q1: Incremental contribution, lack of theoretical validation, and missing experimental time.

A1: For the concern on contribution, we clarified it in the third and fifth paragraphs of the Introduction. The first contribution of our work is a node-level structural entropy metric that quantifies the importance of individual nodes in preserving the global structure. Existing sample selection methods focus primarily on local information and overlook global information, which can result in suboptimal selection. To address this limitation, we explored possible solutions and identified graph structural entropy as a metric that accurately captures the structure of a sample graph. However, it only provides a single value for the whole graph. It is challenging to decompose this metric to the level of individual nodes, limiting its utility for fine-grained, node-level selection. To overcome this, we used the Shapley value to decompose the structural entropy. However, this introduced another challenge: the exponential-time complexity of calculating the Shapley value. To address this, we reformulated the Shapley value and proposed a node-level structural entropy metric that enables linear-time calculation with respect to the edge number. Based on this, our second technical contribution is developing an efficient structural-entropy-based sample selection method that tightly integrates both global and local metrics. The experiment results in Tables 1, 2, 3 of our paper clearly demonstrate that our method consistently improves state-of-the-art methods across all scenarios and tasks. We hope that our work can encourage a closer look at the incorporation of global information into sample selection and potentially spark interest in further exploration.

For the concern on theoretical validation, we have added a theoretical justification and conducted an ablation study to support quantifying node-level structural entropy by Eq. (4). The theoretical justification is provided in Sec. 4.1 and Appendix C. We have proved that $S_e(u)$ provides a lower bound for the sample coverage. This result indicates that maximizing $S_e(u)$ during selection inherently improves sample coverage. Given the strong correlation between sample coverage and the empirical loss of learning algorithms [1], selecting samples with high node-level structural entropy effectively enhances model performance. The ablation study is provided in Appendix G, and the results on CIFAR 10 are as follows. These results demonstrate that removing the second term slightly increases performance across all sampling rates. This finding supports our decision to remove the second term in quantifying node-level structural entropy.

For the concern on missing experimental time, we have included the realistic time for experiments on the CC SBU Align dataset in Sec. 5.1.2. Our method reduces the fine-tuning time on a single Nvidia Tesla V100 GPU from approximately 30 minutes to 3 minutes, adding only a negligible selection overhead of 2 seconds.

Q2: The concern on time complexity.

A2: In our experiments, we empirically found that an appropriate value for $k$ is $\log_2n$, where $n$ is the number of samples. With this choice, the $k$NN graph has $O(n\log_2n)$ edges, resulting in a Shapley value calculation time complexity of $O(n\log_2n)$. This complexity is comparable to sorting and is lower than or equal to that of typical sample selection methods, which is at least $O(n\log_2n)$ when sorting is required and $O(n^2)$ when an enumeration of all pairs of samples is required. We agree with you that for extremely large datasets (e.g. DataComp [2] with billions of samples), this complexity becomes impractical. A potential solution is to split the dataset into bins before selection. Determining an optimal splitting method remains an open question and a promising direction for future research. We clarified this in Appendix E.3.

Q3: Insufficient justification for global and local information.

A3: Thank you for pointing this out. For the justification of global information, measured by node-level structural entropy, we have provided a theoretical justification in Sec. 4.1 and Appendix C. For prioritizing high-difficulty samples (local information), previous research [3, 4] has demonstrated that training difficulty is an effective local metric for quantifying a sample’s impact on model performance. This is because prioritizing high-difficulty samples enables the model to focus on challenging cases, which are typically more informative for improving decision boundaries and generalization [3, 4]. We make this clearer in Sec. 4.2.

Q4: Comparative analysis with methods that prioritize diversity.

A4: Thank you for the constructive suggestion. We have added two widely used baselines that prioritize diversity [5, 6]:

• $k$-means [7], which selects the samples closest to $k$-means clustering centers.
• $k$-DPP [8], which employs a determinantal point process to encourage diversity.

The experimental results are shown in Tables 1, 10, 11, 12, 13, 14, 15, 16 of our paper, and the corresponding discussion is added to Sec. 5.1. These two methods are competitive in low-sampling-rate settings because they ensure the coverage of the dataset. However, in high-sampling-rate settings, they face challenges in balancing diversity with sample importance, leading to suboptimal performance. Our method consistently performs better than these two methods across all datasets and sampling rates. This highlights the importance of balancing the selection of important samples with the preservation of sample diversity.

Q5: Dependence on embedding quality.

A5: We agree with you on the importance of embedding quality for selecting high-quality samples. Indeed, most sample selection methods rely on high-quality embeddings to measure sample similarities, which influence the assessment of diversity and coverage. Improving embedding quality can benefit many such methods, including ours. However, exploring embedding improvement strategies is beyond the scope of our paper. Following common practice, we assumed sufficiently high-quality embeddings are available. In our experiments, we used CLIP and SentenceBERT due to their demonstrated performance in generating high-quality embeddings that capture semantic similarities across various domains. We make this clearer in Sec. 5.1.1.

Q6: Insufficient evaluation for continual learning.

A6: Thank you for pointing this out. Given the diverse scenarios of continual learning and our limited computational resources, we focused on class-incremental learning in this paper, as it is the most representative scenario for continual learning [9]. We make this clearer in Sec. 5.3.1. We have added three widely used baselines on class-incremental learning from prior continual learning studies [10, 11]:

• $k$-center [12], which iteratively selects samples that are least similar to those already selected.
• Gradient Matching [13], which selects samples whose average gradient closely approximates the average gradient of all samples.
• FRCL [14], which optimizes a subset of samples to minimize the posterior uncertainty of the Gaussian process induced from the embedding representations.

We have also added two memory sizes: 50 and 400. The experimental results are shown in Tables 3, 18, 19 of our paper, and the corresponding discussion is added to Sec. 5.3. Our method consistently performs better than the new baselines across all datasets for both the original memory sizes (100 and 200) and the newly added memory sizes (50 and 400). These results validate the effectiveness of our method. We believe that a comprehensive evaluation of our method across various continual learning scenarios would be an interesting direction for future research.

In addition, we have added: 1) two widely used baselines that prioritize diversity (Sec. 5.1) and 2) additional ablation studies to evaluate the quantification of node-level structural entropy, the methods for combining global and local metrics, and the replacement of graphs with hypergraphs (Appendix G).

Q7: Effects of encoding tree construction.

A7: The construction of the encoding tree is indeed an important step in our method. To the best of our knowledge, there are two methods for constructing the encoding tree: the one proposed by Li & Pan [15] and the one proposed by Zhu et al. [16]. Both methods adopt the variations of the Huffman tree to construct the encoding tree. The key difference is that Zhu et al.'s method further compresses the tree to a certain height to improve efficiency in subsequent processing. We assessed the two methods by calculating the Pearson correlation of the resulting node-level structural entropy. The discussion is provided in Appendix A, and the results are as follows. The correlations on CIFAR10, CIFAR100, and ImageNet-1K exceed $0.99$. Therefore, the choice of the construction method has a negligible effect on the sample selection results. In this paper, we employed the more recent method proposed by Zhu et al due to its efficiency.

Q8: Importance score calculation.

A8: In our method, there is a one-to-one correspondence between the data points and the graph nodes. We make this clearer in Sec. 4. Therefore, the importance score is calculated node-wise, which reflects the importance of the corresponding data point. Regarding the second question, since each data point corresponds uniquely to a single node, no data point appears in multiple nodes. This ensures that our method is unaffected by redundancy among nodes.

Q9: Combination methods of global and local metrics.

A9: We explored possible alternative methods to combine the global metric $S_e(u)$ and the local metric $S_t(u)$ other than the proposed multiplication. Specifically, we experimented with the sum ($S_e(u)+S_t(u)$), the harmonic mean ($\frac{S_e(u) S_t(u)}{S_e(u)+S_t(u)}$), and the maximum ($\max(S_e(u),S_t(u))$). The ablation study is provided in Appendix G, and the results on CIFAR10 are as follows. These results demonstrate that multiplication achieves the best average performance, validating our choice for combining the two metrics.

Q10: Theoretical explanation.

A10: The theoretical explanation is provided in Sec. 4.1 and Appendix C. We have proved that $S_e(u)$ provides a lower bound for the sample coverage. This result indicates that maximizing $S_e(u)$ during selection inherently improves sample coverage. Given the strong correlation between sample coverage and the empirical loss of learning algorithms [1], selecting samples with high node-level structural entropy effectively enhances model performance.

References

[1] Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-Centric Coreset Selection for High Pruning Rates. In Proceedings of the International Conference on Learning Representations, 2022.

[2] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. DataComp: In Search of the Next Generation of Multimodal Datasets. In Proceedings of Advances in Neural Information Processing Systems, 2024.

[3] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep Learning on a Data Diet: Finding Important Examples Early in Training. In Proceedings of Advances in Neural Information Processing Systems, pp. 20596–20607, 2021.

[4] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning. In Proceedings of Advances in Neural Information Processing Systems, pp. 19523–19536, 2022.

[5] Ke Shang, Tianye Shu, Hisao Ishibuchi, Yang Nan, and Lie Meng Pang. Benchmarking Large-Scale Subset Selection in Evolutionary Multi-Objective Optimization. Information Sciences, 622:755-770, 2023.

[6] Elisa Celis, Vijay Keswani, Damian Straszak, Amit Deshpande, Tarun Kathuria, and Nisheeth Vishnoi. Fair and diverse DPP-based data summarization. In Proceedings of International Conference on Machine Learning, pp. 716-725, 2018.

[7] Zhao Xu, Kai Yu, Volker Tresp, Xiaowei Xu, and Jizhi Wang. Representative Sampling for Text Classification Using Support Vector Machines. In Proceedings of Advances in Information Retrieval, pp. 393–407, 2003.

[8] Alex Kulesza and Ben Taskar. $k$-DPPs: Fixed-Size Determinantal Point Processes. In Proceedings of the International Conference on Machine Learning, pp. 1193–1200, 2011.

[9] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A Comprehensive Survey of Continual Learning: Theory, Method and Application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362-5383, 2024.

[10] Zal{'a}n Borsos, Mojmir Mutny, and Andreas Krause. Coresets via Bilevel Optimization for Continual Learning and Streaming. In Proceedings of Advances in Neural Information Processing Systems, pp. 14879–14890, 2020.

[11] Jie Hao, Kaiyi Ji, and Mingrui Liu. Bilevel Coreset Selection in Continual Learning: A New Formulation and Algorithm. In Proceedings of Advances in Neural Information Processing Systems, pp. 51026–51049, 2024.

[12] Ozan Sener and Silvio Savarese. Active Learning for Convolutional Neural Networks: A Core-Set Approach. In Proceedings of the International Conference on Learning Representations, 2018.

[13] Trevor Campbell and Tamara Broderick. Automated Scalable Bayesian Inference via Hilbert Coresets. Journal of Machine Learning Research, 20(15):1–38, 2019.

[14] Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional Regularisation for Continual Learning with Gaussian Processes. In Proceedings of International Conference on Learning Representations, 2020.

[15] Angsheng Li and Yicheng Pan. Structural Information and Dynamical Complexity of Networks. IEEE Transactions on Information Theory, 62(6):3290–3339, 2016.

[16] He Zhu, Chong Zhang, Junjie Huang, Junran Wu, and Ke Xu. HiTIN: Hierarchy-aware Tree Isomorphism Network for Hierarchical Text Classification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 7809–7821, 2023.

We hope that our responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript.

We sincerely thank you again for your recognition of our work. As we mentioned in the previous response, we have revised the paper per your suggestions. We hope that these responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript.

If you have any additional comments, we will do our best to address them.

Thank you for your timely feedback and constructive suggestions. We have carefully considered your feedback and conducted additional experiments to address your concerns. Please see our response below.

Regarding the concern on the theoretical justification, we have conducted an empirical verification to show that $\frac{1}{nk^2R}\exp(\frac{1}{kR}S_e(u))$ provides a relatively tight lower bound for $P(u,r)$. In order to precisely calculate $P(u,r)$, we simulate the data using a Gaussian Mixture Model. For this model, the class centers $\mu_1, \dots, \mu_C$ are sampled from a standard Gaussian distribution $\mathcal{N}(0, I)$. Within each class $c$, the samples are drawn from $\mathcal{N}(\mu_c, I)$. To ensure that the synthetic data aligns with the scale of real-world datasets, we match the statistics of CIFAR10 by generating 10 classes with 5,000 samples per class. Our results show that 1) $P(u,r)/\big(\frac{1}{nk^2R}\exp(\frac{1}{kR}S_e(u))\big)$ ranges between $1.00$ and $1.45$ for $90$% of the samples; 2) $P(u,r)/\big(\frac{1}{nk^2R}\exp(\frac{1}{kR}S_e(u))\big)$ is less than $1.97$ for $99$% of the samples. These findings indicate that maximizing $S_e(u)$ during selection improves sample coverage. In the final version of the paper, we will include a detailed discussion of this empirical verification in Appendix A. Nonetheless, we agree that it is important to provide a tighter upper bound with theoretical guarantees. This would be a very promising topic for future research.

Regarding the insufficient experiments, we carefully studied the four suggested papers. Three of them (DER++ [1], GSS [2], and GCR [3]) focus on replay memory construction, while the other one (GEM [4]) focuses on improving the downstream use of the replay memory in model training. We have included the three replay memory construction methods as baselines. Regarding GEM, as we focus on replay memory construction, we did not include it in our experiments. Nonetheless, we appreciate its importance and will discuss how GEM can enhance performance when paired with different construction methods in the final version of the paper. Moreover, we have run the experiments under the task-incremental learning scenario and are in the process of implementing the code necessary for the experiments under the task-free learning scenario. We started to implement and launch the experiments immediately after receiving your feedback. All these experiments are run on 40 GPUs, including 16 NVIDIA A800 GPUs, 8 NVIDIA A100 GPUs, and 16 NVIDIA GeForce RTX 3090 GPUs. According to our estimation, the experiments will take three weeks to complete. As the experiments are time-consuming, we report the initial results, the task-incremental learning scenario on CIFAR10 with a memory size of 100, as follows. In the initial results, our method performs better than the new baselines across learning scenarios. In the final version, we will include DER++, GSS, and GCR as baselines, discuss the use of GEM, and present the complete results in Appendix F.3.

Thanks a lot to the authors for the detailed response to my questions. I will keep my score at this moment, but open to discussion with other reviewers and AC in the discussion phase after rebuttal.

We sincerely thank you for your valuable and helpful suggestions. We are encouraged by your positive feedback and carefully addressed the concern on the theoretical justification. Please see our response below.

Q1: Theoretical justification and discussion.

A1: We have added the theoretical justification and mathematical properties of the node-level structural entropy $S_e(u)$ in Sec. 4.1 and Appendix C. Specifically, we have proved that $S_e(u)$ provides a lower bound for sample coverage. This result indicates that maximizing $S_e(u)$ during selection inherently improves sample coverage. Given the strong correlation between sample coverage and the empirical loss of learning algorithms [1], selecting samples with high node-level structural entropy effectively enhances model performance.

We also empirically demonstrated the effectiveness of our method for quantifying node-level structural entropy through an ablation study. The ablation study is provided in Appendix G, and the results on CIFAR10 are as follows. These results demonstrate that removing the second term slightly increases performance across all sampling rates. This finding supports our decision to remove the second term in quantifying node-level structural entropy.

Reference

[1] Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-Centric Coreset Selection for High Pruning Rates. In Proceedings of the International Conference on Learning Representations, 2022.

We hope that our responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript.

We sincerely thank you again for your recognition of our work. As we mentioned in the previous response, we have revised the paper per your suggestions. We hope that these responses have addressed your concerns satisfactorily. We are grateful for your insightful comments, which have helped us improve the clarity and quality of our manuscript.

If you have any additional comments, we will do our best to address them.