Basel: Target-Aware Basis Selection for Language Models

Since the approach is iterative and gradually removes less relevant parameters, I would expect to see experimental results on how long it takes to adapt a model for the datasets. This could yield important insights on how expensive the Basel technique is. The computational overhead of Basel is comparable to baseline methods, such as SVD and FWSVD. To illustrate, consider compressing the 7B Llama model for a mathematical reasoning task:

The computational overhead of Basel is comparable to baseline methods, such as SVD and FWSVD. To illustrate, consider compressing the 7B Llama model for a mathematical reasoning task:

• SVD requires 3 epochs of fine-tuning, followed by compression, and then 3 epochs of post-compression fine-tuning, totaling 104-128 GPU hours.
• FWSVD requires 3 epochs of fine-tuning, 1 epoch for profiling statistics (which is quite slow), compression, and then 3 epochs of post-compression fine-tuning, totaling 152-176 GPU hours.
• Our method Basel, though initially stated to require a fine-tuned model as input, can effectively use a pre-trained model without performance loss. It involves 3 epochs for compression and 3 epochs for post-compression fine-tuning, totaling 108-132 GPU hours.

Thus, the computational overhead of Basel is within the range of the baseline methods.

 

I would appreciate comparing performance and cost to well-established techniques such as knowledge distillation. I understand that, depending on the complexity of any given dataset, the cost may vary, especially with regard to retraining of the base (l. 209).

While knowledge distillation and low-rank approximation both aim to improve model efficiency, they belong to distinct categories of model compression, each targeting different aspects of optimization. For example, knowledge distillation aligns activation distributions, while low-rank approximation focuses on compressing weight matrices into compact, low-rank forms.

Because these methods address different challenges, direct comparisons may not provide a meaningful evaluation of their respective strengths. That said, techniques from these categories are often complementary and could potentially be combined to achieve even greater efficiency gains. For this paper, we chose to focus exclusively on low-rank approximation to ensure a rigorous and in-depth exploration of its potential.

 

The authors use reasoning benchmarks for their experiments only. What about language understanding (e.g., MMLU)? I would expect the language understanding capabilities of models to decrease, but at what point would this become noticeable?

Our approach is not dependent on the specifics of reasoning or non-reasoning benchmarks, making it applicable to both types of tasks. Furthermore, the benchmarks we use—code generation and math question answering—also require a strong foundation in language understanding. Testing on these benchmarks effectively evaluates our approach's performance in language understanding, as providing correct answers inherently requires a solid grasp of the questions posed.

The novelty of the proposed method is limited. As mentioned by the authors, there are existing works on utilizing singular values in LLMs and pruning based on importance. The technical contributions of this submission are not clear.

We respectfully disagree and highlight the following novel contributions of our work:

• We identify the semantic meanings of bases in LLMs, providing new insights into their interpretability.
• We propose a novel approach to relearn the importance of bases for target applications, rather than simply copying the importance from pretrained or fine-tuned models.

As stated in the related works section (lines 115–117), prior studies prune bases based on singular values derived from pretrained or fine-tuned models without leveraging the training data of the target applications. In contrast, our method explicitly learns the importance of bases using task-specific data, enabling better adaptation to target tasks. This distinction represents a key technical innovation of our approach.

 

The experiments are not convincing and lack some detail. Although the authors include lots of related works, they compare the proposed method against a very limited set of baselines.

We have compared our method against state-of-the-art low-rank decomposition techniques, which are the most relevant baselines for this work.

 

Since Basel includes new trainable parameters (such as scales and additional bases) and is trained on task-specific datasets, it is not surprising that Basel can achieve better performance on these tasks. The conducted comparisons do not convincingly demonstrate the effectiveness of the proposed method.

We would like to clarify a few points:

• The baseline approaches are also trained on task-specific datasets. The improved performance of Bazel is not merely a result of using task-specific training but rather stems from the effectiveness of our proposed method in optimizing base importance for target tasks.
• When comparing our method with baseline approaches, we account for model size, including the new trainable parameters introduced by Bazel. Despite having a smaller overall model size, Bazel consistently achieves higher performance. This result underscores the efficiency of our method and is a notable achievement.

We hope these clarifications address the concerns and highlight the effectiveness of our proposed approach.

Lack of comparison with other model compression methods. Currently, the most common compression method in practical is model quantization, but the experimental part of this paper does not compare with it. A set of comparative experiments with the state-of-art quantization method should be added. On the other hand, model quantization is not task-specific, but Bazel can only be effective for specific tasks, so some additional discussion is needed to explain the necessity of this method.

Low-rank approximation is a distinct category of model compression techniques, fundamentally different from approaches like quantization, weight pruning, and knowledge distillation. Each category leverages unique properties to improve model efficiency: quantization reduces weight precision, pruning exploits weight sparsity, knowledge distillation aligns activation distributions, and low-rank approximation compresses weight matrices into compact, low-rank forms. Since these methods target different aspects of model optimization, direct comparisons across categories are neither necessary nor meaningful. Moreover, techniques from different categories are often complementary and can be combined to achieve further efficiency gains. Therefore, this work focuses exclusively on low-rank approximation to ensure a focused and rigorous evaluation within this category. Regarding task specificity, model quantization can indeed be either task-specific or task-agnostic, depending on the application scenario. Bazel is task-specific, designed for situations where a pretrained model is fine-tuned and compressed for specific application domains. Such scenarios are common in practice; for instance, AR/VR companies often adapt large pretrained models for AR/VR-specific tasks.

 

Lack of experiments on the overhead of performing the proposed compression methodA set of experiments should be added to show the time and resource cost of performing Basel compression

The computational overhead of Basel is comparable to baseline methods, such as SVD and FWSVD. To illustrate, consider compressing the 7B Llama model for a mathematical reasoning task:

• SVD requires 3 epochs of fine-tuning, followed by compression, and then 3 epochs of post-compression fine-tuning, totaling 104-128 GPU hours.
• FWSVD requires 3 epochs of fine-tuning, 1 epoch for profiling statistics (which is quite slow), compression, and then 3 epochs of post-compression fine-tuning, totaling 152-176 GPU hours.
• Our method Basel, though initially stated to require a fine-tuned model as input, can effectively use a pre-trained model without performance loss. It involves 3 epochs for compression and 3 epochs for post-compression fine-tuning, totaling 108-132 GPU hours.

Thus, the computational overhead of Basel is within the range of the baseline methods.

 

Limited scope of application. One of the main advantages of LLM over previous models is its versatility. However, the method proposed in this paper reduces the size of the model by sacrificing the versatility of the model and only focuses on specific tasks. And the pruning of the model also requires The author needs to give a reason for doing so.

Versatility is valuable, but unnecessary versatility can be detrimental, as it introduces significant inference overhead. Bazel addresses this challenge by enabling the adaptation of a pretrained model for one or more specific application domains through compression. This approach is well-suited to practical scenarios, such as AR/VR companies that adapt large pretrained models to develop efficient, domain-specific solutions for their applications. By focusing on these targeted use cases, Bazel offers an effective method for optimizing LLMs for task-specific contexts without attempting to replicate general-purpose capabilities.

Focusing on low-rank compression approaches for comparison: The paper does not compare other compression techniques widely used to compress LLMs, such as quantization and knowledge distillation. Authors could potentially mention them in related works and why they think they are not comparable.

Low-rank approximation is a distinct category of model compression techniques, fundamentally different from approaches like quantization, weight pruning, and knowledge distillation. Each category leverages unique properties to improve model efficiency: quantization reduces weight precision, pruning exploits weight sparsity, knowledge distillation aligns activation distributions, and low-rank approximation compresses weight matrices into compact, low-rank forms. Since these methods target different aspects of model optimization, direct comparisons across categories are neither necessary nor meaningful. Moreover, techniques from different categories are often complementary and can be combined to achieve further efficiency gains. Therefore, this work focuses exclusively on low-rank approximation to ensure a focused and rigorous evaluation within this category.

 

Limited number of tasks and models: This paper could benefit from running more evaluations on different tasks other than mathematical reasoning and code generation. Additionally, it might be good to show that the approach is generalizable to other models of various sizes.

Our paper includes evaluations on two distinct tasks—mathematical reasoning and code generation—spanning four datasets (GSM8K, MATH, HumanEval, and MBPP). Additionally, we test our approach on two models with different sizes, LLaMA-2 7B and LLaMA-2 13B, to assess its generalizability.

While we understand the value of even broader evaluations, expanding experiments indefinitely is infeasible, as one could always request additional tasks or models. We hope the reviewer recognizes the diversity of our current evaluations and agrees that they sufficiently demonstrate the generalizability and effectiveness of our method.

 

Some terms in the equations are not very well defined. For example, tr() in equation(3) is not defined.

The term tr(), used in Equation (3), denotes the trace of a matrix, which is the sum of its diagonal elements. This is a fundamental concept in linear algebra.