EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary Search

Thank you for your feedback, we address your question about stability below:

Average and standard deviations

Across all of our experiments, we observed that EvoPress is extremely robust across different parameter settings, and generally converges to qualitatively similar solutions. To further validate this, for this rebuttal we have performed ablations for the “fast” version (which should have more variance), searching for unstructured sparsity allocation (which has a large number of choices). The results can be found in Appendix C.2 of the revision.

Results show that EvoPress demonstrates strong robustness across different seeds. Specifically, in 16 runs of pruning Llama-3-8B to 70% unstructured sparsity, the method achieves a mean C4 perplexity of 33.82 with a standard deviation of 0.61 (min: 33.06, max: 35.47). In comparison, the configuration identified by the next better method OWL results in a C4 perplexity of 52.32, making the improvements statistically significant. Notably, the standard deviation on hold-out Fineweb-Edu data is even lower, as visualized in Figure 5 of the revision.

We hope this addresses your concern!

We observed that, following our first rebuttal, the reviewer has added a new point to their review:

The theoretical results are somewhat disconnected from other parts. I think having the theoretical results is a bonus and it can be of interest beyond the LLM context. But some exploration of the function form of f in real experiments or some simple simulation example might better demonstrate the significance of the results

We wish to add a clarification relative to this new point:

Specifically, we would like to emphasize that our main theoretical result in Theorem 1 is directly linked to our method’s practical performance: the challenge in the case of LLMs is precisely the fact that we wish to reach an optimal solution with very few evaluation samples, as evaluating candidate LLMs can be extremely expensive.

This link between theory and practice is illustrated in Figure 1, which shows that EvoPress reaches an optimal solution on a real LLM compression instance for which it is feasible to evaluate optimality, in very few evaluations.

More broadly, we note that most prior work, such as ShortGPT, ShortenedLlama, Weight Subcloning, OWL or SPDY, rely on the linearity assumption either implicitly or explicitly. Specifically, linearity is central to the concept of “scoring” the importance of model components, as all the above methods seek to maximize the sum of importance scores for the compressed model. Unlike these methods, the evolutionary algorithm we employ is not restricted to optimizing linear functions and is capable of optimizing diverse fitness landscapes. By proving that EvoPress optimizes efficiently under the linearity assumption (Theorem 1), we provide a theoretical justification for its consistent superior performance across different compression methods when compared to prior scoring-based approaches.

We do acknowledge the reviewer’s point and will clarify this connection further. If the reviewer finds it useful, we can provide additional results on standard “benchmark” functions for evolutionary algorithms, although this direction is further removed from our main scope.

Thank you for your feedback. We address the two points raised below:

“The theoretical justification in Theorem 1 is irrelevant to the algorithm proposed”

We respectfully disagree with the statement that Theorem 1 is irrelevant to the algorithm. First, the vast majority of prior work on non-uniform compression is based on variants of this linearity assumption, either explicitly or implicitly: this is the case for ShortGPT, ShortenedLlama, Weight Subcloning, and OWL, but also dynamic-programming-based approaches such as SPDY, which makes this assumption explicit in their Equation (1). Therefore, showing optimality of our algorithm under this assumption is meaningful.

Second, the linearity assumption is also inherent to the widely used terminology of layer “saliency”. While we don’t argue that the fitness landscape induced by dynamic model compression is entirely linear, it follows from such standard assumptions that it behaves like a linear function to a certain degree, in particular, regarding smoothness. For example, removing the first transformer block of a model has a catastrophic effect on performance. This is the case independent of the compression allocation in other blocks, and can be regarded as assigning an importance measure to this block, resulting in a linear function to be optimized under an l1-constraint. Third, EvoPress uses a general evolutionary algorithm, capable of optimizing a wide range of fitness landscapes. By proving that the entire class of linear functions can be optimized efficiently, we provide a theoretical foundation for the superiority of our approach compared to previous work, which does not provide any guarantees. As also noted in our paper (Footnote 1, Line 107), showing convergence on linear functions is a standard benchmark for evolutionary algorithms, see e.g. (Droste et al., 2002), (Doerr & Ku ̈nnemann, 2015), (Lengler & Spooner, 2015).

Finally, we emphasize that the main contribution of this work is an empirical one. In Figure 1, we demonstrate that EvoPress is able to find the global optima for block dropping search in a constrained setting. Most importantly, we significantly outperform all previous methods–some of which are specialized to specific compression types– across a wide range of compression approaches, models, and compression levels.

The comparison lacks rigor [...] What about results at other sparsity levels?

We understand the importance of a thorough evaluation and would like to clarify that results regarding other sparsity levels were already included in the submission: as stated in line 434, these results for 50% are in Table 13, and for 60% in Table 14. Moreover, we would like to emphasize the thoroughness of our experiments. We cover three different compression approaches, each on three to four levels of compression, on models from three different families. By way of comparison, most previous works limited themselves to a single compression approach.

Thank you for your feedback, we provide answers below:

1. Additional illustration of the method.

Thank you for this suggestion. We added a sketch of the illustration to the Appendix H of the revised version.

2. Hyper-parameter transfer.

The multistep selection is purely for efficiency. Here, the main design decision is how many tokens should be required to have enough certainty to discard the current best search point and adopt the new one. Then, one can design the preselection accordingly, because it comes at little overhead compared to the final selection stage. While we used different parameters for different compression approaches, this was merely to account for differences in the search space / database. For example in unstructured sparsity, the allocation is more fine-grained compared to the quantization bit width allocation, motivating the use of fewer tokens, as more generations will be needed to converge. In the context of unstructured sparsity allocation, we additionally present a “super-fast” version, which uses a more lightweight selection process, and show that it converges to similar validation perplexity (see Figure 3), demonstrating the robustness of EvoPress to these selection hyperparameters.

Regarding the mutation rate, we found the search to perform efficiently as long as the mutation rate is sufficiently small: this is illustrated in Appendix B.1.

Most importantly, we emphasize that we used the same mutation rate across all tested models from different families, across all compression methods, and across all compression levels. We believe this provides sufficient empirical evidence to demonstrate the generalization capabilities of EvoPress. Moreover, in Appendix C.2, we provide results for an ablation running the search for multiple seeds, showing that EvoPress converges to qualitatively similar solutions across different runs and exhibits low variance.

Thank you for the detailed and insightful comments.

We address the questions below, following the numbering in your review:

1. Evolutionary search vs elitist strategy focused on exploitation. We agree with your observation that such an exploitation-focused evolutionary strategy resembles a form of random coordinate ascent. Indeed, these approaches are fundamentally related, though they have been popularized in different scientific communities. Broadly speaking, random coordinate ascent with a different mutation operator (such as ours) can be understood as a form of evolutionary algorithm. Variants of such algorithms are even observable in nature – for example, species that lack crossover and exhibit low population diversity, improving primarily through mutation. Even evolutionary strategies designed to emphasize exploration tend to behave like exploitation-focused searches locally, as broad exploration comes with high computational cost. Consequently, even if the mutation rate is high, meaningful progress is typically driven by offspring with very few mutations, and EvoPress builds upon this observation. Viewing such a zero order search as an evolutionary process offers a more intuitive perspective. For instance, multistep selection can be interpreted as a discrete approximation of the continuous selection observed in natural evolution. Moreover, adapting our approach to slightly different settings becomes straightforward with this abstraction. Framing the method as evolutionary provides greater flexibility, as one can easily adjust the mutation operator or integrate a different underlying evolutionary algorithm.

2. Variability in compression profiles. To fully address this, we included additional ablations on this topic in Appendix C.2 of the revision. Our findings show that the search process is highly robust to different seeds, with the resulting compression profiles being remarkably consistent across runs. This suggests that local optima are generally not a concern.

3. Multi-modal compression cases. To address this concern, we conducted an experiment with joint depth pruning and quantization in the Appendix G of revised revision. Specifically, on each step of the evolutionary algorithm we perform an alternating optimization between block dropping and quantization. Firstly, we select the optimal depth pruning and configuration and then exchange quantization bit-width between “alive” blocks. Our experimental results suggest that EvoPress manages to find a better solution given some starting point and exhibits relatively stable convergence. However, we acknowledge that such a multimodal approach can be inherently more challenging to optimize, with specific outcomes heavily influenced by the choice of compression techniques employed.

4. Step size. The step size per mutation is fixed at one. However, multiple mutations can be applied to the same layer, implicitly allowing for a (slightly) larger step size when generating offspring. We also experimented with sampling step sizes from a different distribution, but did not find this approach to be beneficial. Therefore, for simplicity, we chose to keep the step size fixed.

5. Using additional stages. We employed three selection stages for unstructured sparsity and quantization search, primarily, because we used a higher number of tokens for the final selection decision, and the preselection introduced little overhead. Extending the selection process further has rapidly diminishing returns. Intuitively, after having generated a certain number of offspring, one of them will have higher fitness than the parent. However, the “gains” are not realized when generating further offspring, and it would be more beneficial to generate further offspring by copying this fitter offspring instead of the less fit current parent, i.e. progressing one generation. In general, no matter how many offspring are explored per generation, the algorithm will take a certain number of generations just to move in Hamming distance to the optimum. Hence, we found that investing additional compute per generation is generally not worth it. The “super-fast” version presented for the unstructured sparsity search is a consequence of this observation (it uses two instead of three selection steps). Additionally, reducing the number of tokens in pre-selection is only beneficial up to a certain degree, as evaluation time becomes dominated by the time to load the layers corresponding to an offspring.