Balancing Model Efficiency and Performance: Adaptive Pruner for Long-tailed Data

Thank you for your recognition of our work. We have carefully understood the relevant weaknesses you mentioned and have made the following efforts:

• About broader references

We have incorporated the studies you mentioned into our discussion and references as follows:

Recent advances in long-tailed learning address class imbalance through diverse strategies:

1. Theoretical & Unified Frameworks: [Wang et al., 2023] proposed a data-dependent contraction technique to unify re-weighting and logit adjustment.
2. Hierarchical Label Variation: [Yang et al., 2023] introduced DirMixE, leveraging Dirichlet meta-distributions to model global-local test-time variations, with variance regularization for stable generalization.
3. Ensemble Calibration: [Aimar et al., 2023] formulated BalPoE, a Fisher-consistent ensemble combining logit-adjusted experts calibrated via mixup.
4. Weight Regularization: [Alshammari et al., 2023] demonstrated that simple weight decay and MaxNorm constraints outperform complex rebalancing methods.
5. Contrastive Rebalancing: [Cui et al., 2023] proposed PaCo, integrating parametric class centers into contrastive learning to mitigate gradient bias toward head classes.
6. Margin Adjustment Theory: [Kini et al., 2023] analyzed VS-loss, combining additive/multiplicative logit adjustments to optimize margins in overparameterized regimes.
7. Test-Agnostic Aggregation: [Zhang et al., 2022] developed SADE, a self-supervised expert aggregation framework for unknown test distributions.

These works collectively advance the field by bridging theory-practice gaps [1,6], enhancing model flexibility [2,7], and simplifying regularization [4], while addressing both label and group imbalances [3,5,6].

• About the supplementary experiment on CIFAR-100-LT (IR=10)

According to your suggestion, we supplemented the comparison experiment with IR=10, and the results are shown in the table below.

• About criteria weight matrix

To facilitate your understanding, we have supplemented the images of the weight matrix $D$ in the last training process, where the horizontal axis represents the index of the class, different color blocks represent different pruning weight calculation methods, and the length of different color blocks represents the numerical value of different pruning weights in the weight matrix $D$.

https://anonymous.4open.science/r/AEFCDAISJ/D_matrix_pic.png

• About weight adjustment mechanism

The observations you make are very sharp. In a single step of equation (6), for the improved category $c$, the weights of all relevant criteria indeed increase by the same value $\beta$. We understand that this may raise questions about how the mechanism actually chooses the 'most effective criteria', and we provide the following clarifications in this regard:

(i) Equation (6) is not designed to directly identify a single 'most efficient criterion', but rather to implement a more nuanced and multi-dimensional dynamic balancing mechanism.

(ii) Among other things, the reasons for this design rather than directly identifying the 'best criteria' are:

1. In practice, multiple criteria often work together to achieve the best results, rather than a single criterion being dominant.
2. As reviewer LjiE said, the core goal of this paper lies in introducing distribution-aware capabilities for pruning strategies. The above standard is the carrier of this distribution-aware capability to effective pruning. In future work, we plan to explore mechanisms for more precise and interpretable distribution-aware parameter pruning.

• About novelty

As stated in the previous point, our goal is to introduce distributional awareness to pruning, and to this end, LTAP redefines the pruning problem in the long-tailed scenario from 'uniform compression' to 'differentiated parameter assignment', a conceptual shift that is significantly innovative in its own right. This method of directly correlating distribution properties with parameter importance opens a new path for the application of neural network compression on imbalanced data.

• About notation and spelling errors

We appreciate your comments and will fully revise them in a subsequent version.

Thank you for your recognition of our work. We have carefully understood the relevant weaknesses you mentioned and have made the following efforts:

• About presentation form

Thank you for your valuable suggestions. We will supplement the theorems and textual explanations in Section III with some visual instructions to help the reader understand. We will illustrate (i) why pruning in the long-tail scenario requires special protection, and (ii) the core idea of distribution-aware pruning through a schematic illustration.

• About symbolic expression

Thank you for your advice. We carefully proofread the notations used in this paper and improved some inappropriate descriptions.

Thank you again for your comments and we will continue to work on improving the quality of the manuscript！

Thank you for your time and effort. We are encouraged by your high appreciation of the novelty and contribution of our paper. We will continue to work on exploring how to better solve long-tailed problems in real-world scenarios. If you have any further questions, we will address them at any time. Thanks again!

Thank you for your time, effort and recognition of our work. Your comments are very important for us to continue to improve this work. Based on your comments, we continue to make the following efforts:

• About Pseudocode

This is our negligence. We have completed and corrected the pseudocode in the original paper.

• About the broader baselines

Thank you for your approval of the experimental section. Based on your comments, we tried our best to supplement some baselines in the limited time, including Focal Loss, Logit Adjustment, and RIDE. The following are the experimental results regarding supplemental baseline on the CIFAR-100-LT dataset. The experimental results show the superior performance of LTAP.

• About Figure 2

Thanks for your suggestion, we have adjusted the font of Figure 2 in the original paper to improve readability.

• About Questions

Indeed, the comparison with smaller models is an interesting point, which is fundamental to pruning studies but has unique significance in long-tailed scenarios.

First, as visualizations in Figures 2 to 6 demonstrate, LTAP pruned parameters in a non-uniform, category-aware manner, which is difficult to achieve with manual architecture design. By simultaneously considering multiple importance metrics and adjusting their weights based on accuracy feedback, LTAP may identify subtle parameter interactions that might have been missed in simple architectural reductions. This is fundamentally different from simply reducing the model size uniformly.

In addition, we try our best to organized a contrast experiment: the performance comparison between the manually designed small model and the original standard model, where the flops of the small model is 30% of the original standard model. If needed, you may kindly compare this with Table 1 in the original paper.