Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization

1. The effectiveness of MolPeg is highly dependent on the quality and suitability of the pretrained models utilized.

Thanks for your valuable comments. We agree with the reviewer’s point that the quality of pre-training can influence the effectiveness of our method. In source-free transfer learning, pretrained model is a hard-encoded module, and their variations naturally lead to performance changes. To address the reviewer's concern more thoroughly, we have conducted additional experiments on the HIV dataset using two pretrained models of different quality, obtained from the ZINC-100K and QM9 datasets, respectively. Compared to the PCQM4Mv2 dataset used in the main text, these two datasets are smaller in scale and exhibit more pronounced distribution shifts, resulting in poorer pretraining quality. The experimental results are shown in Table 2 of the PDF file in the General Response.

We observe the following trends: MolPeg still achieves the best performance with these two pretrained models, demonstrating that while pretraining quality is a key factor affecting performance, MolPeg remains the most robust and effective method compared to existing DP strategies. We again thank the reviewer for pointing out this issue and hope this explanation addresses your concerns.

2. The proposed framework requires the maintenance and simultaneous updating of two models, which could increase both implementation complexity and computational overhead compared to simpler pruning methods.

Thanks for your valuable comments. The complexity is undoubtedly crucial for efficient learning, but we respectfully disagree with the reviewer regarding our method being more costly compared to existing pruning methods. It is worth noting that most existing DP methods are static and require scoring and ranking on the full dataset before training, which involves a high cost and computational overhead. We address the reviewer's concerns from two perspectives: complexity analysis and experimental validation.

• Complexity Analysis: To compare the computational complexity of our method, MolPeg, with other data pruning methods, we use the following notations: $N$ is the total number of data points, $\delta$ is the pruning ratio, $T$ is the total number of training epochs, $t$ is the number of pre-scoring epochs required by other methods to determine scores, and $H_{fw}, H_{bw}$ denotes complexity of forward and backward operation. Note that compared to the TopK operation, the majority of the time consumption comes from the process of scoring the samples, as this process typically involves computing the loss, gradients, or even more complex metrics. Therefore, we have disregarded the complexity of the sorting process and focus on the forward and backward complexity.

  	MolPeg	Other methods
  Forward passes	$\mathcal{O}(2H_{fw}(N))$	$\mathcal{O}(H_{fw}((t + T\delta)N))$
  Backward passes	$\mathcal{O}(H_{bw}(T\delta N))$	$\mathcal{O}(H_{bw}((t + T\delta)N))$

  While MolPeg performs more forward passes due to processing two models as noticed by the reviewer, the computational bottleneck is the backward pass, which typically takes more than two times the effort of a forward pass [1]. Other methods require $tN$ extra backward passes for pre-scoring, which outweighs their advantage in forward passes.

  Therefore, despite the added forward pass complexity, MolPeg is overall more efficient due to the significantly reduced number of backward passes.

  [1] PyTorch Distributed: Experiences on Accelerating Data Parallel Training. VLDB 2020.

• Experimental Validation: In section 5.1 of the manuscript, we have provided an experimental analysis of efficiency comparison. As seen in Figure 4 of the manuscript, our method demonstrates significantly shorter runtime compared to previous DP methods and achieves better model performance. Compared to the latest dynamic pruning methods, although our runtime efficiency is slightly behind, we achieve better generalization performance, which is crucial for transfer learning scenarios. This minor efficiency compromise is acceptable.

3. Molecular datasets represent a more specialized field. It is crucial to test MolPeg with mainstream validation datasets to further demonstrate its effectiveness.

Thanks for your constructive suggestion and helpful feedback. To further demonstrate MolPeg's effectiveness, we have provided the experimental results on CIFAR-10 and CIFAR-100 in Table 3 of the rebuttal PDF file. For the experimental setup, we follow the settings of InfoBatch. However, since our method requires a pretrained model, we adopt MoCo strategy to pretrain the ResNet-18 on ImageNet, and then fine-tune it on the CIFAR dataset. As seen from the experimental results, MolPeg still achieves the best results on most pruning ratios, further validating the effectiveness of our method across different domains.

Regarding the reviewer's concern about the used datasets, we would like to reclaim our research motivation. As mentioned in General Response, our research is problem-driven and targets on ubiquitous and unsolved problem in AI for Chemistry community. Moreover, molecular tasks represent a complex and comprehensive task system. Unlike traditional DP tasks in CV, which focus on image classification, we validate MolPeg on both classification and regression tasks, involving various molecular modalities and diverse task types, which makes our evaluation even more comprehensive than traditional DP approaches. We will add the above experimental results to the appendix of the revised manuscript and we hope this response addresses your concern.

Dear Reviewer uU9Z,

During our proofreading process, we discovered a typing error in the time complexity of the forward passes for the MolPeg method. It should be $\mathcal{O}(2H_{fw}(T\delta N))$, while the textual analysis in the initial rebuttal is correct. We sincerely apologize for this oversight and invite you to refer to the corrected time complexity table below:

We appreciate your understanding and attention to this correction.

Best regards,

Authors

(Minor) Missing recent works: 1) static data pruning [1,2], 2) dynamic data pruning [3]

Thanks for your recognition of our work and constructive suggestions. We apologize for omitting any relevant works. For the related works provided by the reviewer, we have added experimental results on the HIV and PCBA datasets for the first two works as additional baselines (AL and CCS). The experimental results are shown below:

It can be observed that MolPeG still achieves the best performance with significant improvements. We will add the above experimental results and empirical analysis to the appendix of the revised manuscript to enrich our research.

For the last related work, due to the time constraint during the rebuttal period, we are unable to reproduce their results. However, we will also include it in the discussion of related works in the revised version. Finally, we appreciate your helpful comments. If you have any other questions, please feel free to let us know.

1. Although the paper claims to be pioneering in applying data pruning to pretrained models, the motivation may require further exploration. Specifically, how can it ensure that OOD samples crucial for each task are not pruned, which could potentially undermine the very purpose of the pretrained model?

Thanks for your valuable feedback and insightful concerns. Preserving crucial samples is indeed a challenge for static DP methods in OOD scenarios. However, we would like to clarify that we employ a dynamic DP strategy. We invite the reviewer to refer to the General Response describing our pruning setup and the pseudo-code in the Appendix G for a better understanding. Below, we explain in detail why OOD samples crucial for each task are not pruned:

• Broader Receptive Field of Dynamic Pruning Strategy. Unlike static DP, although we use a fixed proportion of samples in each iteration, we monitor the training dynamics of the entire dataset. Our method naturally avoids the issue of completely ignoring crucial samples, as almost all samples are used to varying degrees. Coordinating the use of samples in each iteration to achieve better generalization is the main contribution of MolPeg. In Figure 1 of additional PDF file, we visualize the frequency of sample usage in the HIV datasets, showing that almost all samples are used for training even with aggressive pruning ratio, with crucial samples being used more frequently.
• Crucial OOD samples are essentially our hard samples. Note that it is a contentious issue to determine whether or not a sample is crucial. Even in supervised training with the full dataset, the importance of different samples vary across epochs. In our scenario, crucial OOD samples can be considered as the ones that the training struggles with, corresponding to those with gradients opposite to the EMA gradient, as theoretically analyzed in Proposition 2. Therefore, we actually regard crucial OOD samples as an important category (hard cases) to preserve, rather than filtering them out.

2. The definition of what constitutes a 'hard case' is unclear. Are these cases task-specific? If so, there's a risk that pruning might eliminate hard cases essential for certain tasks, affecting the model's comprehensiveness and utility.

We apologize for any lack of clarity in our descriptions. Below, we provide clearer explanations for hard and easy cases to address reviewer's concern, and we will also include these in the revised main text for better understanding.

• Hard cases refer to samples that the model struggles with during optimization. These can also be understood as samples near the decision boundary of downstream tasks. As analyzed in lines 156-160 of our manuscript, these samples satisfy $\mathcal{L}(x,\theta_t)-\mathcal{L}(x,\xi_t)>0$，indicating that this epoch's optimization gave negative feedback compared to historical optimization.
• Conversely, easy cases are samples that can be optimized very smoothly, leading to a continuous loss reduction, satisfying $\mathcal{L}(x,\theta_t)-\mathcal{L}(x,\xi_t)<0$.

Moreover, we want to emphasize that neither hard nor easy cases are pre-defined; they are identified in real-time based on the loss discrepancy reflecting training dynamics, making them definitely task-specific since the loss value is closely related to the specific task. Regarding the reviewer's concern about the risk that eliminating hard cases, we believe this is similar to concern in Weakness 1. In our previous response, hard samples are actually crucial OOD samples. Therefore, our method does not eliminate these cases but rather preserves them as informative ones.

3. The approach might exacerbate the 'molecular cliff' problem, where slight changes in molecular structure lead to significant changes in activity, and such nuances could be lost with aggressive data pruning in pre-training.

Thank you for your constructive feedback. We believe there might be some misunderstanding regarding our data pruning setup. As elaborated in the General Response, our data pruning is conducted on the downstream dataset, not during the pretraining stage.

Furthermore, we would like to address the reviewer's concerns from an experimental perspective. If our method exacerbated the molecular cliff phenomenon in the pruned training set, the test performance in a random split setup would be poor. This is because retaining only these special cases during training would lead to a significant distribution shift between the training and test sets. However, the experimental results in Section 5 demonstrate SOTA data pruning performance, and these experiments were all conducted under the random split as mentioned in line 232 of our manuscript. Therefore, we believe that the issue the reviewer is concerned about does not arise with our method.

4. The experimental section is limited to only three commonly used molecular datasets. Other equally important datasets, such as MUV, were not included in the comparisons.

Thanks for your valuable comments to further enrich our empirical analysis. We have supplemented our work with additional experiments on the MUV dataset, following the same experimental setup described in the manuscript. Please refer to Table 1 in the PDF file in the General Response for the results. We observe that MolPeg still achieves state-of-the-art performance on the MUV dataset, further validating the effectiveness of our method. In the revised version, we will include these additional experimental results in the appendix to enrich the experimental validation.

5. It is suggested that the discussion of limitations be moved to the main text.

Thank you for pointing out this problem. In the revised version, we will follow the reviewer's suggestion and move the limitation to the main text.

Hi Reviewer zkte,

Does the author’s response address your concerns? Please acknowledge that you have read the responses at your earliest convenience.

Best wishes,

AC