PruneFuse: Efficient Data Selection via Weight Pruning and Network Fusion

We appreciate the reviewers’ feedback and the opportunity to address their concerns. We have responded to each comment below to provide additional clarity.

1. Confusion about Figure 2. Using actual trajectories would make this figure clearer.

Figure 2 provides a conceptual visualization of the proposed framework, illustrating how pruning and fusion reshape the optimization dynamics and improve convergence. It is not intended to depict actual training trajectories. For empirical evidence, we direct the reviewer to Figure 5, where the training trajectories are shown. The results in Figure 5 clearly demonstrate that the proposed model fusion achieves faster convergence and better accuracy due to improved initialization of the network.

2. The initialization process seems to be unstable.

Pruning from Scratch [1] demonstrates that pruning at initialization not only reduces training time but also provides robust solutions by enabling the exploration of sparse architectures that generalize well. In our experiments, we observed consistent and stable training behavior across multiple runs with different random seeds, confirming that the initialization process does not introduce instability. Specifically, the pruned networks exhibit predictable performance trends: as the pruning rate increases, the accuracy of the network decreases proportionally. Furthermore, we observe that data selection quality and the overall performance of PruneFuse improve when the pruned network retains more parameters, underscoring the robustness of our initialization process. These empirical results and theoretical insights from prior work validate the stability of the initialization process used in our method.

3. The Baseline in Table 2 doesn't correspond to each other between Tsync = 1 and Tsync = 2 when label budget is 50%, which also unmatch the data in Table 1.

Thank you for pointing this out. We made a typo in Table 2 (Tysnc=1 for 50% budget), the value should be 93.61%. We have fixed it in the revised manuscript. Since Tsync is a parameter specific to PruneFuse V2, the baseline results remain the same for Tsync=1 and Tsync=2.

Additionally, as Table 1 reports results for PruneFuse V1 and Table 2 focuses on PruneFuse V2, the baseline methodology was slightly adjusted for fairness. Specifically, in Table 2 when compared against the PruneFuseV2, which uses the trained fused model to guide the data selector, we modified the baseline to continue retraining the network from the previous round rather than reinitializing it (also mentioned in the results section 5.2). This adjustment led to slight differences in baseline accuracy and ensures a consistent and fair comparison within the context of PruneFuse V2.

4. The motivation for using pruned networks is not very clear, as it seems other teacher-student models with similar structures can achieve comparable effects.

The primary motivation for our approach is to enhance the efficiency of active learning pipelines, particularly in resource-constrained environments, while maintaining a generic and adaptable framework. Handcrafted teacher-student models, often require significant manual effort to design and are not easily adaptable to varying computational or task-specific demands. As demonstrated in comparison with SVP (Fig. 4 in the main paper and Table 13 in the Supplementary Materials), such models may not always achieve optimal compatibility or performance.

Additionally, the student models used as surrogates in traditional teacher-student frameworks are typically being discarded after data selection, as there is no systematic way to integrate the insights gained during their training back into the teacher model. In contrast, our approach provides a systematic and generic method for designing data selectors through pruning and fusion, allowing us to utilize the knowledge from the trained selector models to improve the accuracy and efficiency of the final models. The proposed pipeline is scalable, adaptable, and applicable across diverse scenarios and computational settings, addressing the challenges and inefficiencies associated with teacher-student models while providing a more robust and reusable framework.

Thank you for the thorough evaluation of our paper. We value the detailed feedback and have provided clarifications to each of the comments below.

1. Concerns regarding the computational costs when neural network pruning typically has high training costs. Also, the pruned network is also trained for fusion, which increases the training costs.

The pruning process is performed once before training, involving a single computation of L2 norms and sorting, with a complexity of $O(P$ $log P)$. The fusion process, which integrates the weights from the trained pruned network $\theta_p^*$ into the original network $\theta$, is a lightweight operation with a complexity of $O(P)$, introducing negligible overhead. The detailed complexity analysis is provided in the Supplementary Materials section A.1. The primary computational effort lies in training the pruned network for data selection, which, particularly in the context of active learning, is performed over multiple rounds. In PruneFuse, the reduced size of the pruned network ensures significant efficiency compared to training the full network in each round, as demonstrated in Figure 3 of the main paper. Additionally, the fusion process accelerates convergence during the subsequent training of the full model, as illustrated in Figure 5. Furthermore, we have now included runtime comparison of PruneFuse versus the baseline in detail in Table 19 and 20 of the Supplementary Materials, further underscoring the practical efficiency of our approach. Together, these results reinforce the computational effectiveness and scalability of the proposed methodology.

2. Selection methods like Moderate and CCS have low computation costs. Report and compare the training costs of each module.

The suggested techniques, Moderate [1] (which selects data points closer to the distance median from a class center) and CCS [2] (which improves data coverage for coreset selection), are primarily scoring strategies rather than computational optimizations for the data selection pipeline. Both methods require training the same model on which their scoring is based, resulting in computational costs comparable to other data selection strategies. That being said, these scoring strategies can be seamlessly integrated into the proposed PruneFuse pipeline. For a comprehensive evaluation, we have incorporated Moderate and CCS into our framework and present the results in Table 17 of the Supplementary Materials. The experiments demonstrate that while these strategies provide alternative scoring mechanisms, the computational efficiency and overall performance of PruneFuse remain superior due to the reduced size of the pruned network and the efficiency of the fusion process. These findings highlight the adaptability of PruneFuse and its ability to integrate diverse scoring strategies while maintaining its computational and performance advantages.

3. How is figure 2 drawn? Clarify the meaning of different colors.

Figure 2 conceptually illustrates the evolution of training trajectories under the proposed framework. The contours represent the loss landscape, with colors transitioning from red (higher loss) to blue (lower loss). Subfigure 2a shows the trajectory of the original network $\theta$ in its unmodified loss landscape. After pruning, the landscape is tailored, as shown in 2b going from yellow (high loss) to blue (lower loss), simplifying the optimization process for the pruned network $\theta_p$, which converges to an optimal point denoted as $\theta_p^*$. Subfigure 2c demonstrates the refined trajectory of the fused model $\theta_F$, which benefits from the initialization provided by $\theta_p^*$, achieving a superior trajectory and improved convergence in the original landscape.

This conceptual illustration aligns with the empirical results in Figure 5, where the faster convergence of the fused model $\theta_F$ is clearly demonstrated. Together, these figures emphasize the role of pruning in reshaping the optimization dynamics and the advantages introduced by the fusion process.

Thank you for your detailed and thoughtful response to my questions. While some of my concerns have been addressed, several critical issues remain:

• Regarding Q1: Tables 19 and 20 indicate that the training costs of the data selectors are nearly 50% or more of the target model training costs. When combined with the marginal improvements in accuracy, the practical significance of the approach appears limited. Furthermore, comparing the selection costs solely against the baseline and target model training is not entirely fair, as these methods often have higher costs. It is strongly recommended that the authors compare the actual selection costs against other state-of-the-art (SOTA) methods for a more detailed evaluation.

• Performance comparison: In Table 1, the main performance comparison only involves the baseline model, which is insufficient to validate the proposed method’s effectiveness. Although the method can integrate scoring mechanisms, it lacks comparisons with more advanced SOTA methods, which is not convincing.

• Regarding Q8: Including more advanced deep learning models, such as Vision Transformers (ViT), would further demonstrate the effectiveness of the proposed method.

• The references and methods used for comparison are outdated.

• I have concerns about the generalization of the proposed method across different architectures, i.e., can datasets selected by one trained pruned network generalize well to other networks? This is a crucial issue for data selection methods, as it is impractical to select subsets tailored to every possible model that may be used in the future.

Since some critical concerns are not addressed, I will maintain my score.

Dear Authors,

Thank you for your detailed response. I will take the response into account and listen to other reviewers' voices during the AC-reviewer discussion phase and decide whether to raise my rating.

Best regards

We thank the reviewer for the thoughtful feedback on our paper and appreciate the opportunity to address the concerns raised in detail below.

1. Pruning without training leads to straight removal of layers. Are there existing studies to support this strategy.

The approach of pruning from a randomly initialized network, as employed in our work, is well-supported by prior research. Specifically, the Pruning from Scratch [1] demonstrates that effective pruned structures can emerge directly from randomly initialized weights without requiring pretraining. This method broadens the search space for optimal architectures, unlike pruning pre-trained networks, which are inherently biased by their initial training trajectory. Moreover, the Lottery Ticket Hypothesis [2] further supports the feasibility of identifying sparse, trainable subnetworks at initialization. In our experiments, pruned models derived from randomly initialized weights consistently demonstrated robust data selection capabilities and effective fusion with the original network. This empirical evidence, combined with the findings of [1] and [2], validates the soundness of our pruning strategy at initialization. Additionally, we explore alternatives to the static initial pruning, such as iterative pruning in PruneFuseV2 (Section 4.6), where the trained fused network generates a pruned network for subsequent data selection. Furthermore, we include results for dynamic pruning, where pruning occurs progressively over the first 20 epochs (Table 18 in the Supplementary Materials), to evaluate the impact of different pruning methodologies. These additional experiments demonstrate the flexibility of the proposed framework and its ability to effectively incorporate various pruning strategies.

2. Ablation study on the use of knowledge distillation.

PruneFuse demonstrates strong performance even without incorporating knowledge distillation (KD). However, KD is integrated into the framework to provide additional optimization for the fused model $\theta_F$. By reusing the logits from the pruned model $\theta_p^*$, which are readily available from its training phase, KD is incorporated without incurring any additional computational overhead.

Detailed ablation studies on KD, presented in the Table 10 of the Supplementary Materials, confirm its modest contribution. KD marginally improves performance, particularly in high-label budgets (e.g., $b = 40$%). However, the core performance gains of PruneFuse stem from the proposed model fusion, which significantly enhances both efficiency and convergence.

3. Performance improvements seems marginal and a direct runtime comparison would be insightful.

The primary motivation for our work is to make the time-intensive routine of active learning more efficient, particularly in resource-constrained environments. To support this, we have included detailed runtime comparisons in Table 19 of the Supplementary Materials, which evaluate the computational efficiency of our method across various architectures and datasets. Our results demonstrate that PruneFuse achieves substantial reductions in computational overhead compared to baseline methods.

4. Comparison with recent works.

While we recognize the value of recent works, we specifically chose SVP [15] as our primary baseline due to its direct comparability and versatility. We reference many recent works in Section 2, including SubSelNet (NeurIPS 2023) [3]; however, SubSelNet requires a computationally intensive pre-training routine on a large pool of architectures and this process must be repeated for any change in data or model distribution. Such demands can be impractical, particularly in resource-constrained or dynamic environments. In contrast, SVP is a more practical and effective benchmark for data selection, which is why we chose it for comparison.

To provide a broader evaluation, we also implemented three recent coreset selection techniques, Forgetting-events [4], Moderate [5] and CSS [6], and present a detailed comparison in Table 17 of the Supplementary Materials. These results demonstrate that PruneFuse remains effective when combined with these advanced scoring techniques, maintaining its computational efficiency while achieving strong data selection performance across diverse scenarios. This further establishes PruneFuse as a robust and adaptable framework for active learning.