EvoPress: Accurate Dynamic Model Compression via Evolutionary Search

We thank the reviewer for a thorough and detailed view with a lot of insightful comments and suggestions. The concerns are addressed below:

Stopping Criteria

We ran the search for more generations than necessary to get insights about the convergence behavior. We agree that in practice one should determine the number of generations dynamically, and as suggested, we have implemented a stopping criteria: Every 20 generations the perplexity on the training set is computed, and if it hasn’t improved, the search terminates. We present timings and results of EvoPress including early stopping below.

Runtime vs. baselines

Determining meaningful runtime results for baselines is challenging due to several reasons:

• For block dropping, 3 out of 4 baseline methods neither shared their code nor specified data usage, making runtime comparisons difficult. Generally, scoring methods are faster than EvoPress. We compare the runtime of EvoPress with baseline methods below.
• For quantization and unstructured pruning, runtime depends on the specific method, while our approach works independently of the layerwise compression method. OWL, for instance, requires an extensive hyperparameter sweep (20 configurations) and a larger layer database than EvoPress, leading to similar runtime constraints in our SparseGPT setup, dominated by layer database generation.
• Additionally, for pruning, the sweep in OWL is not provided by the authors, and it is not clear how the “best” model is chosen (in particular, over how much data). We decided (in their favor) to evaluate all 20 models on the entire evaluation data, and choose the model with lowest perplexities.

We will make this more transparent in the revised version. To address the reviewer’s concern, we have included runtimes under common parameter values below.

Comparison with baselines

Block dropping

We provide comparisons with SLEB and BlockPruner in Table. We also included the runtimes of other baseline methods, where we used 64K calibration tokens of Fineweb Edu consistently. One can observe that EvoPress outperforms these additional baselines, especially at higher compression, while being faster. We report the results of EvoPress with early stopping as outlined above.

Unstructured sparsity

Plug and Play and Wanda are relevant to unstructured pruning, and we'll include them in the new revision. Our work primarily focuses on finding the optimal sparsification profile on top of a layerwise compression method. We adopted SparseGPT due to its popularity and good performance. We demonstrate that EvoPress also functions effectively on top of a layer database generated using Wanda in Table, where we also report runtime measurements. As requested by Reviewer eQfQ, the table contains results for BESA.

Quantization

AWQ, BiLLM, and OmniQuant will be discussed in our updated version, acknowledging their relevance. To test EvoPress's compatibility with various quantization methods, we applied it to a database generated by AWQ. Due to AWQ's constraints, the search space requires the same bitwidth within transformer block. Despite this, EvoPress achieved an improved compression profile (see Table).

Human-Crafted Scoring

We wanted to emphasize the fact that the baseline methods do not evaluate the fully pruned model, instead, they make strong assumptions on how the fully pruned model performs based on another metric, which is inherent to scoring methods. In contrast, EvoPress (and [1,2] in a more restricted setting) evaluate the fully pruned models directly. We will rephrase this in the revised version.

Questions

• Use of SparseGPT. Regarding SparseGPT, we used it for its reliable performance at high compression, superior to Wanda at 60%-70% sparsity, which are our targets. Wanda, however, allows re-use of weight configurations across sparsities with separate masks.
• Mixed-quantization baselines. At the time of submission, we were not aware of any baselines that address the problem of mixed-precision quantization in a comparable setting. Reviewer eQfQ suggested the Slim-LLM paper, which we compared but found it complementary due to its channel-wise approach. OWL's layerwise non-uniform quantization performed worse than uniform quantization, so we did not include it.
• Convergence Plots of EvoPress. The train accuracy refers to the KL-Divergence of the survivor of the selection process, i.e. the parent of the subsequent iteration. It is measured on a random subset of the full training data (Fineweb-Edu), which is why the line is noisy. The test accuracy is measured on the full set of hold-out Fineweb-Edu data and does not impact the search process. We will clarify this in the revision.

We thank the reviewer for their comments. Below, we address the two weaknesses:

The theoretical proof does not fully explain its effectiveness in nonlinear LLM compression. It is unclear whether EvoPress maintains optimization capability in nonlinear regions.

Fundamentally this is true – we cannot expect the linearity assumption to fully hold in practice. Notably, assuming linearity is standard in prior work, such as OWL, ShortGPT, and DP-based methods such as SPDY. In contrast, while our theoretical results rely on this assumption, EvoPress can optimize a much larger class of fitness environments. In this sense, EvoPress operates under significantly weaker assumptions than prior approaches. That said, our main contribution is empirical, with EvoPress showing strong performance across various settings.

Additionally, we conducted experiments to verify the capabilities of EvoPress:

• In Figure 1, we consider the problem of removing twelve transformer blocks of Llama-3-8B, with the additional constraint that only pairs of consecutive blocks can be removed. We brute forced the entire search space (16 choose 6 = 8008 configurations) using significant compute in order to identify the global optima. Then, we ran our evolutionary approach on this problem, and found that it detects the global optima within 6 generations.
• We performed a robustness analysis for pruning Llama-3-8B to 70% sparsity. We found that over 16 independent runs the resulting perplexity on hold-out data has very little variance (Figure 7), and the final configurations show high similarity (Figure 6). This indicates that local optima do not pose a problem for the search process.

The proposed strategy may get trapped in local optima in high-dimensional and multimodal compressed spaces. Its performance in highly constrained or multi-objective settings remains uncertain.

We would like to point out that the search space is relatively high-dimensional - the dimensionality of space is several dozens for the case of block dropping and several hundreds for quantization and unstructured sparsity. For a high enough compression ratio the loss landscape is nonlinear (as the example in Table 1 shows), yet the evolutionary search procedure successfully converges to a good optimum. EvoPress is compatible with multiple compression techniques simultaneously. Specifically, we conducted an experiment with joint depth pruning (with 25% blocks dropped) and quantization (with 4 bits on average given 2, 3, 4, 5, 6 bit width options) (see details in response to reviewer Q9Fo).

We thank the reviewer for the feedback. The concerns are addressed below:

Experimental design and analyzes

Runtime of the method

Similar as for unstructured pruning in the main text, we have indicated the runtime of EvoPress for block dropping and quantization within convergence plots (see Appendix Figure 8 for Block Dropping, and Appendix Figure 14 for Quantization). We provide runtime measurements in the additional baseline comparisons further down.

You correctly observed that the super-fast version depicted in Figure 3 does not fully reach the model performance of the more heavy-weight search. However, the hit in performance is rather small, and we wanted to demonstrate that it can be obtained for a fraction of the search time. This also suggests a practical strategy: one could first run the faster search to quickly reach good performance, and only then switch to the more heavy-weight search for best results.

Comparison with Baselines

We acknowledge your point; below, we provide additional comparisons with existing methods.

1. Depth pruning

We acknowledge that SLEB is a relevant baseline and will include it as related work in the revision. Below, we have included a comparison with EvoPress on Llama2-7B, where we also included BlockPruner as a baseline, following the suggestion of Reviewer zC2u. Both SLEB and BlockPruner iteratively remove blocks and can thus be interpreted as highly restricted search procedures. In such methods, once a block is removed, it cannot be recovered in later iterations, which explains the worse performance compared to EvoPress. We report the runtime of EvoPress when using a simple stopping criteria, that terminates the search whenever the perplexity on the training data has not improved within the last 20 generations. The comparison of different depth pruning methods is detailed in Table.

2. Unstructured sparsity

Indeed, BESA is a relevant work and will be added to the related work section in the revised edition. For a fair comparison, we ran the fast version of EvoPress on a Wanda layer database (thus, the results are slightly worse than presented in the paper, where we used SparseGPT). Still, EvoPress finds notably improved layerwise sparsity allocations.

For completeness, we also included the row-wise version of BESA, which adapts the Wanda Mask within each layer. We show that this is complementary to EvoPress by using the average per-block sparsity found by EvoPress as the per-block sparsity for BESA (EvoPress+BESA), which mostly yields better perplexities than BESA and EvoPress individually. However, this is still very much experimental, and there might be better methods to merge both approaches, like producing a layer database using BESA with row-wise non-uniform sparsity allocation, and then searching this database with EvoPress.

The experimental results with BESA in the original setup as well EvoPress applied on top of BESA and Wanda are provided in Table.

3. Quantization

Slim-LLM and CWMQ perform mixed precision quantization on a per-channel level compared to the per-layer precision allocation adopted in our method. Therefore, we could adopt Slim-LLM or CWMQ as an alternative to GPTQ, and run EvoPress on top of this method. A direct comparison between EvoPress and these two works is not possible. However, we believe that EvoPress would synergize very well with such intra-layer mixed precision quantization methods, as it allows to produce layers with non-integer bitwidth, which is beneficial for the search procedure. According to your suggestion, we will reference these two works in the updated revision.

Slim-LLM

We conducted Slim-LLM experiments using the open-sourced implementation on GitHub. We considered two setups:

• Original calibration data (Wikitext2, 128k samples of length 2048)
• Our calibration data (FinewebEdu, 8M tokens)

EvoPress shows significant improvement over SlimLLM in the paper's original setup in terms of 0-shot and MMLU evaluations. When comparing methods with the same calibration data, both methods show similar performance. However, the runtime of Slim-LLM increases dramatically with more data (~50 hours on single RTX3090 GPU compared to ~10 hours for EvoPress). That said, EvoPress and SlimLLM are complementary and hence, running EvoPress on top of SlimLLM will lead to further gains. We present results in Table.

CWMQ

The algorithm implementation of CMWQ is not open-sourced, making it difficult to provide a comparison in a comparable setup.

We thank the reviewer for the feedback. The questions are addressed below:

To quantify model degradation, KL divergence was used in Section 3. KL divergence is popular to do it, but there are other ways, for example max absolute value, or we can even use a small calibration dataset to measure perplexity, etc. It’ll be great if authors could explain this design choice.

In our work we considered both KL-Divergence and Perplexity on the calibration set as fitness functions in the evolutionary search. We provide an ablation on these choices in Appendix B.3. Both choices show similar performance, but we decided to stick to KL-Divergence as it performed slightly better. The comparison between these two options is provided in Table 1.

Table 1. Comparison between KL-Divergence and Perplexity as the fitness function

We believe this is because KL-Divergence measures relative degradation to the base model, which is more informative than a pure next token loss. This is especially important when using as little calibration data as we do in our multistep selection process. We decided to adopt KL-Divergence to quantify the distance between the two predictive distributions, as it is a well-established metric, with a grounding in information theory.

The paper has a high level problem definition. However, the proposed approach was only applied to single compression approach one by one (pruning or sparsity or quantization). It will be great if authors could show or explain the results of applying EvoPress on multiple compression approaches simultaneously.

EvoPress is compatible with multiple compression approaches simultaneously. We conducted an experiment with joint depth pruning (with 25% blocks dropped) and quantization (with 4 bits on average given 2, 3, 4, 5, 6 bit width options). Specifically, on each step of the evolutionary algorithm we perform an alternating optimization between block dropping and quantization. Firstly, we select the optimal depth pruning and configuration and then exchange quantization bit-width between “alive” blocks. Our experimental results (see Wikitext-2 perplexity and C4 perplexity) suggest that EvoPress manages to find a better solution given some starting point and exhibits relatively stable convergence.