Thank you so much for your detailed comments and suggestions, our response is as follows.

Weakness 1. Limited comparison with existing compression methods and baselines.

The comparison would be more comprehensive by comparing with more baselines which are train-free and trained.

Thanks for the suggestion. However, we think training is too heavy for low-end devices, since memory usage can be three to four times as much as inference [21].

We have added SVD-based compression as our baseline in General Response G3.

Weakness 2. Evaluation is limited on small GPT models.

Evaluation focused mainly on GPT-family models and mainly focus on small models.

We have extended our experiments to OPT series models, please refer to General Response G3.

Regarding the small model size, please refer to General Response G1.

Weakness 3. Compability with other model compression approaches.

It would be a great to combine the proposed embedding compression method with other model compression methods to check compatibility.

Thanks for the suggestion; we are considering quantization and plan to take it as future work.

Question 1. Language models outside GPT family.

We have extended to OPT, please refer to General Response G3.

Question 2. Could this approach compatible with other model compression methods?

Yes, quantization is the easiest one, and it is orthogonal to our approach.

The combinations with weight tying mentioned by Reviewer JBLc, and pruning like that in SliceGPT[6] are a bit difficult, but we believe it can be solved by changing product/multiplication sequences.

Question 3. Have you investigated the effect on model robustness like in the multilingual setting using more diverse tokens?.

Thanks for the suggestion. At the moment, we suspect that the multilingual tasks are too complex for the small language models, as we discussed in Tab.xaFK.3, Weakness 3 with Reviewer xaFK. MGSM [19] in Tab.xaFK.3 is also a multilingual dataset, but all the investigated sub-billion models cannot perform well. We feel that it is better to evaluate robustness only when the language models perform well on the task.

However, we acknowledge that multilingual tasks are rather valuable for the robustness of the edge application, and plan to take this as our future work. Yes, quantization is the easiest one and is orthogonal to our approach.

Reference

[19] "Language models are multilingual chain-of-thought reasoners." arXiv preprint arXiv:2210.03057 (2022).

[21] Zhao, Jiawei, et al. "Galore: Memory-efficient llm training by gradient low-rank projection." arXiv preprint arXiv:2403.03507 (2024).

Thank you so much for your comments, our response is as follows.

Weakness 1. The novelty is limited.

Tensor-Train Decomposition is explored in several language model compression works. What are the differences between the proposed method and existing works?

As far as we know, [11,16,17] are the only works that use Tensor-Train Decomposition for the language model compression. All of them require extra training, which is unrealistic for low-end devices. Our training-free approach is suitable for meeting the requirements of adaptivity and low energy in edge applications, as we discussed in General Response G0.

We would highly appreciate it if the Reviewer were aware of such work and possibly gave more references for it, so we could discuss them in our paper.

Weakness 2. Lack important baselines.

Compressing token embedding layers is studied in some papers and the paper does not compare with them. For example, [1] and baselines used in [1] should be set as baselines.

Thanks for the reference. We have discussed the works about token embedding layer compression ([7,10-12, 14, 17]) in the General Response G2. Given that we only focus on small language models deployed on low-end devices, only [10] solves the same problem as ours and should be compared. However, [10] still requires training a meta-model, which should be fine-tuned when the deployment environment changes. So, we only take the SVD-based approach, the same as that in LightToken [7] (reference [1] in the Reviewer's original reviews), as our baseline. The experimental comparison of this part is in the General Response G3.

Weakness 3. Limited evaluation.

The empirical evaluation is primarily on language modelling and sentiment classification, with potential limitations in representing other complex NLP tasks, such as math reasoning and multi-hop question answering.

Thanks for pointing out. We report the results of zero-shot reasoning in General Response G3.

We suspect that math reasoning and multi-hop question answering, which are still hard problems for LLMs, are too complex for sub-billion language models on low-end devices.

We have evaluated some sub-billion language models on MGSM [19] for math reasoning and DROP [20] for multi-hop question answering, and it came out that all the models could not perform well. Our view is that there is no need to compress models for these two tasks on low-end devices.

Tab. xaFK.3 Small language model scores on math reasoning (MGSM) and multi-hop question answering (DROP)

Reference

[7] "LightToken: A task and model-agnostic lightweight token embedding framework for pre-trained language models." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.

[10] "Direction is what you need: improving word embedding compression in large language models." arXiv preprint arXiv:2106.08181 (2021).

[11] "Tensorized embedding layers for efficient model compression." arXiv preprint arXiv:1901.10787 (2019).

[17] "Efficient gpt model pre-training using tensor train matrix representation." arXiv preprint arXiv:2306.02697 (2023).

[19] "Language models are multilingual chain-of-thought reasoners." arXiv preprint arXiv:2210.03057 (2022).

[20] "DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs." arXiv preprint arXiv:1903.00161 (2019).

Thanks for your comments, our response is as follows.

Weakness 1. Overclaim

The authors claim "- As far as we know, we are the first to compress LLMs with low-rank factorization, specifically", which is not true to me.

Thanks for pointing out. We acknowledge that it is confusing and misleading here. We wanted to convey "As far as we know, we are the first to compress Small Language Models (SLMs) [9] for low-end devices use cases, with low-rank factorization'', which has been updated in line 78-80 of the updated submission.

The architecture is the same for most NLP models and thus pardon me I really don't know what's added.

We guess the Reviewer wanted to express that our approach has no obvious contribution in algorithms, since model compression is to work on the models with existing architectures rather than creating a new architecture.

For a short answer, our approach is specifically designed for small language models on low-end devices, to meet the requirements of adaptivity and low energy in edge applications. We systematically analysed energy, latency, etc, which common LLM compression rarely considers.

For a detailed answer, please refer to the General Response G0, G2 and our answers to Weakness 4 raised by Reviewer JBLc.

Weakness 2. About Tensor-Train

I think all the tensor-train stuff has been developed and applied before.

General Response G1 includes all the relevant tensor-train stuff [11,16,17] we know. All [11,16,17] contain extra training, which is unsuitable for low-end devices. In the updated submission, this part has been emphasized in line 189-193.

We would highly appreciate it if the Reviewer were aware of such work and possibly gave more references for such work, so that we could discuss them in our paper.

Weakness 4. Conclusion

It's not immediately obvious to me what's new conclusions or findings drawn from this paper.

Conclusion: our approach is suitable for low-end devices in edge applications.

Question

Pardon me that I really don't know what's the biggest contribution. Other than your claim of try things on LLM, what's the real technical contribution in your paper not discussed by previous works?

As we mentioned in the response to Weakness 2, our contribution is the first work for low-end devices to design a low-rank model compression strategy to satisfy the adaptivity and low energy requirements in edge applications.

For the comparison between relevant works and ours, please refer to General Response G2.

Reference

[8] "MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases", ICML 2024

[9] "Small Language Models: Survey, Measurements, and Insights." arXiv preprint arXiv:2409.15790 (2024).

[11] "Tensorized embedding layers for efficient model compression." arXiv preprint arXiv:1901.10787 (2019).

[17] "Efficient gpt model pre-training using tensor train matrix representation." arXiv preprint arXiv:2306.02697 (2023).

(1/2)

Thank you so much for the very detailed comments and references! Our response is as follows.

Weakness 1. Limited Novelty

... The methodological advancements or generalizations beyond TT_SVD appear minimal based on the methodology section.

Though we did not change the exact implementation of TT-SVD, we did adjust its working unit (vectors rather than the whole matrix) and workflow to cope with the unique issues (adaptivity and low energy, as stated in General Response G0) for low-end devices. We also have a comprehensive systematic analysis regarding latency, energy and flops, and the impacts (latency and accuracy) of the tensor orders in the experimental section.

Also, we do not believe a simple methodology necessarily means no novelty. "Attention is All You Need" and "LoRA: Low-rank Adaptation of Large Language Models" are two examples. Their methodologies are rather simple (as stated in their abstract or introduction), but both revolutionised the community.

Weakness 2. Unconvincing Experiments:

The experiments are conducted on older and relatively small models, such as GPT-2 and models up to only 1.3B parameters...

Please refer to General Response G1, we only focus on SLMs on low-end device.

... the evaluation setup lacks modern benchmarking practices...LLM harness [2] ...

Thank you so much for the helpful evaluation resources, our new experiment results are shown in General Response G3.

Weaknesses 3. Unintuitive Rationale:

The foundational rationale for embedding a matrix using a tensor is unclear. The method requires reshaping the matrix into a higher-order tensor before applying tensor decomposition, but the paper does not provide an intuitive explanation for why this approach is effective or reasonable.

Tensors can model implicit high-dimensional representations (as well as the interactions among orders) of the model weights. In this sense, tensors have better expressivity than matrices, which is also a good solution for small-size models to express complex functionality with limited parameter space.

The results in G3 have empirically proved this point, as tensor-based approaches have higher chances of retaining the language task performance than the matrix-based approach.

... the embedding matrix only occupies a small part of the model even for a mid size of model, and a common way to reduce parameter size is through weight tie, which directly reduce the parameter sizes of embedding into half. How does you method performs in this case?

Firstly, apart from reasoning and language modelling, classification is also an important task for edge applications, which we have investigated in the paper. Weight tying cannot be compiled with the classification layer.

Secondly, we acknowledge that weight tying is a common approach to reducing memory, but we do not reckon it is a necessary part of LMs. Especially for the adaptivity requirements in edge applications (Section 2.1 in the updated submission), there should be a different approach to compress the fully connected layers (i.e. further amplifying the signal from the transformer) rather than directly reusing the weights of the embedding layer.

Weakness 4. Overstatement of Contribution:

Some claims appear overstated. For instance, the authors state, "As far as we know, we are the first to compress LLMs with low-rank factorization, specifically for low-end devices."

Thanks for pointing out. We acknowledge it is confusing and misleading. We wanted to convey "As far as we know, we are the first to compress Small Language Models (SLMs) for low-end devices use cases, with low-rank factorization'', which has been updated in line 82-83 of the updated submission.

However, the cited reference [3] already demonstrates the compression of LLMs using SVD for similar purposes. Furthermore, there are numerous existing works that apply low-rank decomposition for LLM compression, such as [4-6].

Thanks for the references. [3-6] are discussed in General Response G2.

Furthermore, we want to clarify the inconsistency of the term "LLMs". The use of the term "LLM'' in the original submission is to follow the term used in MobileLLM [8]. To avoid confusion, we emphasized that we only focus on sub-billion parameter models in the abstract and the introduction of our original submission. We have updated the paper title in our new submission accordingly, TensorSLM: Sub-billion Parameter Language Model Compression for Low-end Devices based on Tensor-Train Decomposition. This clarifies our focus is not "LLMs" but Small Language Models (SLMs) [9].

(3/3)

Tab.JBLc.3. Zero-shot performance of OPT-1.3B after compression.

We also give the results of SparseGPT. Since SparseGPT only freezes the weights rather than "deletes" them, we do not list its parameter ratio. From Tab Tab.JBLc.2 and Tab.JBLc.3, we can observe that the different compression approaches have different superiorities for different zero-shot reasoning tasks and compression ratios. Even SVD-based sometimes outperforms the others.

These results further indicate that compression approaches of LLMs may not be easily migrated to SLMs, as we discussed in (1/3).

References

[6] "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

[22] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "LLM-Pruner: On the Structural Pruning of Large Language Models." Version v1. May 19 (2023). arXiv:2305.11627.

[23] Men, Xin, et al. "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect." Version v1. March 6 (2024). arXiv:2403.03853.

[24] Frantar, Elias, and Dan Alistarh. "Sparsegpt: Massive language models can be accurately pruned in one-shot." International Conference on Machine Learning. PMLR, 2023.

(1/3)

Thank you so much for your reply and new references for LLM-Pruner [22] and ShortGPT [23]. Our responses to your reply are as follows:

1. Compression of small language models (SLMs) is less general than for large language models (LLMs).

LLMs are more widely used, ...

We acknowledge that "LLMs are more widely used" at the moment, but we do think SLM compression is also important. SLMs are for on-device applications (e.g. mobile phones and Raspberry Pi), and are suitable for applications without stable networks, sufficient GPU resources or continuous power charging. The compression of SLMs directly impacts battery life and, therefore, the user experience.

... and their compression techniques can often be applied to smaller models, whereas the reverse is not always true.

There has yet to be a consensus that the compression approaches of LLMs can be easily migrated to SLMs. On the contrary, there exists empirical evidence that LLM compression approaches cannot maintain the accuracy of SLMs as they can for LLMs:

1. In Tab.2 of ShortGPT [23], for the same model series (Llama2 and Baichuan2), 13B models maintain more model accuracy than 7B models with ShortGPT, as summarized as follows

Tab.JBLc.1. Average score degradation after compression with ShortGPT.

2. As stated in the second last paragraph of the introduction of SparseGPT [24], "larger models are more compressible". In Fig.2 of [24], with the same compression settings, the models with fewer parameters have a more severe accuracy drop (larger slope in Fig.2).

3. In line 418-420 and line 445-448 of our current submission, we addressed that our compression approach performs better on the larger-sized models, which can be observed from Fig. 3 (a-b,d-j,m) in our current submission.

For instance, methods like LLM-Pruner [1] and layer pruning strategies ShortGPT [2] demonstrate fast and effective compression, ...

Thanks for this information. In the referred LLM-Pruner (7B as their smallest tested models) and ShortGPT (2.8B as their smallest tested models) in detail, we did not find their results on sub-billion parameter models. As we stated in line 58 of our current submission, running an uncompressed Gemma-2B on Raspberry Pi leads to a system crash. Thus we only consider models around or less than 1B.

... , even for large models.

Our paper only focuses on SLMs running on low-end devices, which normally cannot hold LLMs. Thus we do not consider the "large models".

(2/3)

2. Lack of comparisons with state-of-the-art (SOTA) methods.

The additional comparisons with SVD are insufficient to justify the proposed method. SVD performs significantly worse compared to other SOTA approaches, making it an inadequate baseline.

Thanks for this emphasis. To find an appropriate baseline, we investigated the references in our General Response, the newly referred LLM-Pruner[22], ShortGPT[23] and a commonly used baseline SparseGPT[24]. Among these references, only SliceGPT[6] is training-free and compresses the embedding layers. The comparisons are in Tab.JBLc.2 and Tab.JBLc.3. Given that SliceGPT also compresses other layers, we only listed the results of similar overall parameter ratios after the compression. The bold numbers are the best performance for each parameter ratio setting.

Tab.JBLc.2. Zero-shot performance of OPT-125M after compression.

General Response (6/6)

Reference

[1] "Tensor-train decomposition". SIAM Journal on Scientific Computing, 33(5):2295–2317, 2011. doi: 10.1137/090752286.

[2] "A framework for few-shot language model evaluation." Version v0. 0.1. Sept 10 (2021): 8-9.

[3] "Language model compression with weighted low-rank factorization." ICLR 2022.

[4] "Asvd: Activation-aware singular value decomposition for compressing large language models." arXiv preprint arXiv:2312.05821 (2023).

[5] "MoDeGPT: Modular Decomposition for Large Language Model Compression." arXiv preprint arXiv:2408.09632 (2024).

[6] "Slicegpt: Compress large language models by deleting rows and columns." arXiv preprint arXiv:2401.15024 (2024).

[7] "LightToken: A task and model-agnostic lightweight token embedding framework for pre-trained language models." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.

[8] "MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases", ICML 2024

[9] "Small Language Models: Survey, Measurements, and Insights." arXiv preprint arXiv:2409.15790 (2024).

[10] "Direction is what you need: improving word embedding compression in large language models." arXiv preprint arXiv:2106.08181 (2021).

[11] "Tensorized embedding layers for efficient model compression." arXiv preprint arXiv:1901.10787 (2019).

[12] "Groupreduce: Block-wise low-rank approximation for neural language model shrinking." Advances in Neural Information Processing Systems 31 (2018).

[13] "Improved Residual Vector Quantization for High-dimensional Approximate Nearest Neighbor Search." arXiv preprint arXiv:1509.05195 (2015).

[14] "Learning k-way d-dimensional discrete codes for compact embedding representations." International Conference on Machine Learning. PMLR, 2018.

[15] "Monarch: Expressive structured matrices for efficient and accurate training." International Conference on Machine Learning. PMLR, 2022.

[16] "Compute Better Spent: Replacing Dense Layers with Structured Matrices." arXiv preprint arXiv:2406.06248 (2024).

[17] "Efficient gpt model pre-training using tensor train matrix representation." arXiv preprint arXiv:2306.02697 (2023).

[18] "Addition is all you need for energy-efficient language models." arXiv preprint arXiv:2410.00907 (2024).

General Response (4/6)

Tab. G.3.3 OPT-1.3B

Tab. G.3.4 CerebrasGPT-111M

Tab. G.3.5 CerebrasGPT-256M

General Response (3/6)

G3. Evaluation of more complex language tasks and LMs outside the GPT family is required.

We have extended our experiments with OPT-{125M, 350M, 1.3B} on zero-shot reasoning tasks with APIs in [1], since OPT performs well on the zero-shot reasoning tasks (as shown in Fig.1 of the updated submission). The experimental results are as follows. The bold numbers indicate the top-3 best performance cases. We also evaluated CerebrasGPT on these tasks, which is available in our updated submission.

We can observe from Tab. G.3.1 - G.3.7 that our approaches have a higher chance of maintaining the language task performance (especially in the average scores).

Tab. G.3.1 OPT-125M

Tab. G.3.2 OPT-350M

General Response (2/6)

G2. Lack of the comparison of relevant works.

We did address our differences with the relevant works in line 42 - 51 in our original submission. Along with the referred works addressed by the Reviewers, the comparison is as following:

Tab.G.2 Study on LM compression or relevant low-rank factorization

It should be noticed that though SliceGPT [6] also works with matrices, it performs a kind of pruning that exploits the sparsity rather than low-rank. Thus, it is outside the scope of low-rank factorization and the relevant works.

From Tab.G.2 we can observe that none of the relevant works has the same focus as ours. Though [10-12, 15-16] do not require fine-tuning the compressed model, they need to train a meta-model to get the compressed weights. If the input token distribution changes, the meta-models still require fine-tuning. Thus, for edge applications in this paper, the meta-learning based approaches [10-12, 15-16] are outside our scope of "training-free".