Structural Pruning of Large Language Models via Neural Architecture Search

We thank the reviewer for the constructive feedback.

1. To address their concerns, we updated our paper and replaced the term ‘large language model’ by ‘pre-trained language model’. Even though LLM with prompt-classification are increasingly popular, fine-tuning models such as BERT is still highly competitive on labeled datasets and often used in practice. We see the extension of our approach to LLM as a promising future work.
2. We tried to benchmark against COFI using the official implementation but obtained inconclusive results compared to the original paper. We decided not to include these results yet to avoid an unfair comparison that would put COFI at a disadvantage. We reached out to the authors to understand this in detail and ensure a fair comparison; if possible, will included the results of experiments for the camera ready version. However, we would like to stress that including additional structural pruning methods would not change the main argument of the paper: NAS allows for multi-objective optimization rather than constrained optimization to obtain the Pareto set of sub-networks. This captures the non-linear relationship between performance and compression ratio. For example, Figure 5 right shows that pruning the model on the MNLI by 50% leads to the same performance as pruning by 60%. Since we do not know this before, we either have to run the constraint optimization process (e.g. COFI or RFP) multiple times based on a grid of threshold values or stick to a sub-optimal solution based on an arbitrarily defined threshold. Multi-objective NAS on the other hand allows us to select the model without any additional overhead.
3. Regarding the callout on comparison with distillation methods (NAS-BERT), we appreciate the feedback from the reviewer and included a more detailed discussion about the differences of distillation and pruning approaches in the related literature section. As we discuss in the related work section and also in the response to reviewer t4jT, in this setting, given a pre-trained model, the goal is to train a smaller model from scratch by distilling the knowledge of a larger pre-trained model. While distillation allows for flexible architectures, as is the case for NAS-BERT, its cost is in the same ballpark as pre-training the teacher model. In our case, we focus on pruning directly on the target task, which is the more appealing scenario for practitioners who do not have the compute to distill their own model and would rather fine-tune a pre-trained model. This makes it hard to perform a fair apple-to-apple comparison between these two approaches. In addition, the two approaches are complementary to each other, since we can always prune a distilled model.
4. Question regarding runtime: For all methods we compute the total runtime to obtain a Pareto set of solutions. This includes also fine-tuning the original model for RFP and running it multiple times with different thresholds.

We thank the reviewer for the thoughtful feedback and we are glad that the reviewer appreciated the experimental evidence of the effectiveness of our approach. We completely agree that compressing pre-trained language models is a crucial research direction for NAS. Indeed distillation techniques such as AdaBERT and TinyBERT are interesting alternatives to utilize NAS. However, there are several reasons why an experiment against distillation approaches would not be an apple-to-apple comparison:

• We tackle the multi-objective setting where we obtain a Pareto set as a solution. Distillation on the other hand only provides a single solution. Providing a Pareto set of solutions is one of the key contributions of our work.
• Distillation requires an expensive offline phase to train a single student model, which can then be fine-tuned on target tasks. Our pruning approach does incur an additional overhead compared to conventional fine-tuning on the target task, but omits the expensive distilation phase; pruning is substantially faster than distillation. Both approaches are valid but it depends on how many downstream tasks are considers (# tuning per experiment) to decide which approach is more useful. This makes it hard to compare the overall required compute time of both approaches.

As suggested by the reviewer, we added a detailed discussion of AdaBERT and TinyBERT to the related work section of our paper and expanded the discussion on differences between pruning and distillation methods. We already discuss NAS-BERT in our paper, which is orthogonal to our approach. Since our method is also independent of whether the network was pre-trained or distilled from another model, it is straightforward to combine it with methods such as NAS-BERT.

We thank the reviewer for the thoughtful feedback and appreciate that they found the paper to be ‘very interesting’ and that they appreciate the in-depth ablation study of the paper.

Inference Time: Could the reviewer clarify if they mean model inference time or the overall runtime of our method? We only report model inference time in Figure 11 in the appendix, where in fact the sub-networks found by our NAS approach are substantially faster than the quantized original network. Decreasing model inference time is one of the key benefits of our method and we use the network size as a proxy for infererence latency throughout the paper. We have added the following sentence in the results section to make it clear - note that parameter count is related to model inference time as discussed in Appendix.

In the case of runtime, our weight-sharing based NAS approach indeed induces some additional time overhead compared to just fine-tuning the model. However, we would like to point out that this time is significantly smaller (see Figure 5 left) than standard NAS or dropping layers. Our multi-objective approach returns a set of solutions such that the optimal compressed network can be selected post-hoc. This is in contrast to other structural pruning approaches, such as layer dropping or RFP, which require a pre-defined threshold on the remaining parameters. Since there is no simple relationship between pruning ratio (i.e., number of parameters left in the model) and performance, to find the optimal sub-network, we have to run these methods multiple times with different thresholds. Having said that, we do think there is room for further research to make weight-sharing based NAS more time efficient, for example by using non-uniform sampling of sub-networks during the fine-tuning of the super-network.

Question Figure 4: We’d like to ask the reviewer to clarify why it seems that more than 1 datasets dropped more than 5% when pruning 50% of parameters. Our experiments show that a performance drop higher than 5% was actually only observed on the COLA dataset.