BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments

As the authors, we sincerely appreciate any time and effort the reviewer has taken to improve our work and are committed to addressing any concerns about the paper to the best of our ability. However, we strongly suspect that this review may have been generated by an LLM, as many of the questions raised appear to be based on misunderstandings or are even irrelevant to our proposed method. Nevertheless, we will do our best to address them here:

Q1.A: The rationale for sorting SVD matrices is not well-explained.

A1.A: Thank you for pointing this out, and we are willing to improve the readability of the paper. We explained the sorting process in Section 2.2 and provided the pseudocode of the sorting process in Algorithm 1, Section A.1 (also referenced in the main paper). We would appreciate if you could specify which aspects of the sorting process require further clarification, allowing us to provide more detailed explanations. In addition, to clarify, we sort "residual blocks," which consist of a packed sign matrix and singular vectors, as illustrated in Figure 3.

Q1.B: The process of loading/offloading within and between layers needs better elucidation.

A1.B: The process of loading/offloading residual blocks does not happen within or between layers, rather, it happens between RAM and storage devices. As illustrated in Figure 2, the residual blocks for each weight stack are loaded/offloaded to the universal stack in the storage in a presorted order, we use the same color for blocks belonging to the same weight stack for better elucidation.

Q2: The paper doesn't provide details about the caching process, which seems to be an integral part of the method. Discussion on cache size and caching strategy is missing.

A2: There is no “caching process” mentioned in any part of the paper. This question seems like a hallucination by an LLM.

Q3: The impacts of SVD on model performance aren't discussed. The paper should elaborate on the effects of various decomposition parameters and distinguish the proposed SVD from those used in previous model compression techniques.

A3: Thank you for raising this point. We address your concerns as follows:

• In BitStack, model performance is not directly influenced by SVD itself as we iteratively perform SVD on the absolute value of the approximation error of each weight matrix(as discussed in Section 2.1). It is the number of loaded residual blocks that directly impacts model performance, which we can control to implement a trade-off between memory consumption and performance. Figure 1a provides an overview of this trade-off, demonstrating that BitStack can recover the full potential of the original model with the same memory consumption while maintaining good performance at lower memory budgets. Additionally, Figure 8 shows that this trade-off is fine-grained, where even a small increase in memory allocation leads to performance improvements.
• Considering the number of decomposition iterations $n$ is forward-compatible(as we perform the decomposition on residuals), the only parameter related to the decomposition is the number of retained singular vectors $k$, which we already did the ablation in Section 3.3 (Figure 7b). The results show that BitStack is largely insensitive to this parameter.
• The main differences between the use of SVD in BitStack and in previous methods can be summarized as follows:
  • 1. BitStack does not incorporate additional processes for the weight scaling before SVD (calculation of the Hessian matrix in FWSVD[1], whitening in SVD-LLM[2], etc.).
  • 2. BitStack performs SVD on the previous approximation errors, instead of directly on the model weights.
  • 3. BitStack retains the sign matrix(which can be packed for further memory reduction) and performs SVD on the absolute value of the approximation errors.

Thank you for your response. To clarify, BitStack is proposed as a general weight compression method for LLMs, and there is no caching involved at the method level. Regarding the restoration of weights from the ITERATIVE ABSOLUTE VALUE DECOMPOSITION, we provide the implementation here:

w = torch.zeros((out_features, in_features))
for i in n_blocks:
		w += unpack(packed_sign) * (u_k @ vt_k)

We believe there is no additional loading should be worried about since we are only performing inference computations with the blocks that are already loaded. During the recovery of the weights, the intermediate results(an accumulator) occupy the same space as the resulting weights, it is unlikely that offloading to storage would happen here(if that is your concern).

Additionally, could you please confirm whether your other concerns from the previous review have been addressed? Please let us know if you have any other questions.

We sincerely appreciate the time and effort you took to review our paper. We are grateful for your recognition of our work, and we acknowledge that most of your concerns are focused on the evaluation aspects. Below, we address these points in detail:

Q1: The evaluations should be enhanced. Both AWQ and GPTQ are no longer considered state-of-the-art quantization methods as of the current time. Even AWQ was submitted to arXiv over a year ago (June 2023).

A1: We acknowledge that both AWQ and GPTQ were proposed over a year ago, and they may no longer be considered state-of-the-art quantization-based approaches. However, we would like to emphasize several important reasons for selecting these methods as baselines:

1. Training-free setting: We consider the experiments in a training-free setting, which is a key advantage for real-world deployment scenarios. Being training-free facilitates fast, low-resource compression, and allows for easy adaptation to various models. We believe this is the reason why GPTQ and AWQ are the only weight compression methods well integrated into commonly used LLM serving engines like vLLM[1] and LMDeploy[2]. Most recent quantization-based methods with stronger performance incorporate further fine-tuning to maintain performance at high compression ratios [3][4][5]. On the other hand, GPTQ and AWQ are still considered very strong baselines in the training-free setting, and recent training-free approaches still perform the comparison with these baselines, for example, DAQ(Oct 2024, [6]) and GWQ(Nov 2024, [7]).
2. Bridging the gap between decomposition-based method and quantization-based method: As you depicted in “Strength”, BitStack is the first decomposition-based method that achieves comparable performance to quantization approaches. We consider this a huge step as previous decomposition-based methods usually compare their performance with plain SVD on LLMs[8][9][10], which leads to significant performance degradation, and can not reach high compression ratios like quantization-based methods (as we depicted in Section 3.1.1). So we choose to compare with practical quantization-based methods to verify the potential of BitStack in practical deployment instead of just comparing to other decomposition-based methods. Nevertheless, we reproduce the results of ASVD[8] and SVD-LLM[10], and compare the perplexity of the Llama 2 7B model on the WikiText2 test set at different compression ratios:

It can be seen from the table that BitStack outperforms previous decomposition-based methods by a significant margin. Furthermore, it is important to note that these SVD-based baselines compute the compression ratio differently from those of BitStack.

Specifically, the compression ratio in SVD-based methods is derived from the concatenated ratio of singular values, while BitStack’s compression ratio is calculated as (total_memory-compressed_model_memory)/total_memory. This calculation takes into account the memory consumption of uncompressed weights in layers such as embedding and layer norm, which is not considered in the SVD-based methods. For example, an SVD-LLM checkpoint at 0.8 compression ratio consumes 5.44GiB memory while BitStack consumes only 2.51GiB. Given these differences, this is even an unfair comparison for BitStack, further demonstrating the effectiveness of BitStack. We have added these evaluations in the revision of the paper(Section A.6).

We greatly appreciate your careful review of our paper and the insightful feedback you've provided. We are glad to see that the novelty and practical utility of BitStack were well recognized. We are grateful for your constructive suggestions, below are our responses to the key comments raised:

Q1: Comparison in efficiency to other memory management techniques.

A1: Thank you for this insightful suggestion. We greatly appreciate your constructive feedback, which allows us to enhance the quality of our paper. We agree that comparing BitStack’s efficiency with other system-level memory management techniques would offer valuable context. Specifically, while most off-the-shelf LLM serving engines support paging only for KV cache (e.g., vLLM), they do not typically offload model weights. To address this, we compare the efficiency of BitStack with the Hugging Face accelerate library, which uses a memory-mapping strategy for offloading model weights [1].

The main difference between BitStack and weight offloading is, given an available memory budget, BitStack adjusts its compression ratio to fit into the available memory and inference with the compressed model; while memory offloading methods do inference with all model weights and have to load/offload every part of the model during each inference.

To illustrate this difference, we present a comparison of the throughput (tokens/s) for Llama 2 7B under various memory budgets. Notably, Llama 2 7B requires approximately 12.55GiB of memory to load fully.

As the table shows, BitStack consistently outperforms model offloading in scenarios where available memory is less than 12.55GiB (i.e., when offloading is required). The throughput of BitStack is higher by a factor ranging from 1.6x to 61.4x, depending on the memory budget. This demonstrates that BitStack provides a more efficient solution for memory-constrained environments, as it eliminates the need for frequent model weight offloading, which would otherwise incur significant performance penalties. Note that these results were measured on a node with H800 GPUs, which offer high memory bandwidth. The latency caused by loading and unloading model weights could be even more pronounced on edge devices with lower memory bandwidths, further emphasizing the advantages of BitStack in memory-constrained environments.

We have incorporated these results into Section A.5.2 in the revised paper. Once again, we appreciate your valuable suggestion!

Q2: In Table 1, 2,3 it might be a good idea to make the best method's results in bold, to make it easier for the reader.

A2: We appreciate your suggestion to highlight the best results in Tables 1, 2, and 3. We have updated the tables in the revision of the paper.

Thank you for your detailed and thoughtful feedback on our paper. We appreciate your positive comments on the clarity of the paper and the practical usefulness of our method, BitStack. We also value your constructive questions, and we would like to address them as follows:

Q1.A: How long does it take to dynamically load/off-load a residual block?

A1.A: The time required to load or off-load a residual block depends on the specific hardware configuration. However, we decompose the weights into residual blocks with minimal sizes (as shown in Section A.4, Table 6). For example, the maximum size of a residual block in an 8B model is 7.56 megabytes. In contrast, the memory bandwidth of current devices is measured in gigabytes per second (e.g., 8GiB/s for the Snapdragon 8 Gen 2 and 6.4GiB/s for the Apple A17 Pro [1]). Therefore, the time taken to load or off-load a residual block should be negligible, especially when compared to the time required to load an entire quantized model at a difference compression ratio, which operates at the gigabyte scale.

Q1.B: What is the strategy to achieve the best performance/latency tradeoff?

A1.B: BitStack mainly implements a trade-off between memory and model performance(quality), we recommend using as much available memory as possible to ensure the best possible model quality, leveraging the flexibility of BitStack. If latency is taken into account, since the BitStack model incurs higher latency when using more memory (more residual blocks are unpacked on the fly), we recommend setting a memory range for BitStack models based on the task difficulty in real-world scenarios. For instance, when handling simple tasks like text summarization, limit the model to using less memory to achieve lower latency; while for more complex tasks, such as complex reasoning, allow the model to use more memory for better performance. For more details on the latency/throughput of BitStack, please refer to Section A.5 in the revised version of the paper.

Q1.C: Is more or less adjustment preferred during inference?

A1.C: Adjustment during inference is encouraged, as the flexibility of BitStack is what sets it apart from other methods. By making dynamic adjustments, we can maximize the utilization of available memory, which ultimately leads to improved model performance.

Q2: Based on the comparison between 8B and 70B, it seems that one should use llama-8B at FP16 instead of llama-70B at an extreme compression ratio. At least in this case, the extreme quantization does not seem to be needed. And I assume that the latency of a quantized larger model is also worse than the latency of a smaller model at FP16.

A2: You are correct that the 8B model at FP16 outperforms the 70B model at extreme compression ratios with lower latency. In fact, this holds true for most weight compression methods, including the baselines discussed in the paper, as well as other well-established methods such as OmniQuant[2], despite it not being training-free. You can refer to [3], which provides comprehensive empirical results comparing a wide range of quantization methods, all of which face this issue, referred to as Pareto Optimal in the AQLM paper[4]. Some recent state-of-the-art quantization methods report higher performance for extremely compressed larger models compared to smaller models at full precision. However, these methods all involve further fine-tuning after compression and complicate vector quantization[4][5][6] at each predefined compression ratio, whereas BitStack is training-free and only requires performing compression once. One typical situation where one would use a 70B BitStack model over an 8B one is when the total memory is relatively large, allowing the 70B model to fully leverage its potential when available memory is sufficient.

We hope the responses above have addressed your concerns. Please feel free to raise any further questions if anything remains unclear. We look forward to your reply! :)

[1] https://nanoreview.net/en/soc-compare/qualcomm-snapdragon-8-gen-2-vs-apple-a17-pro

[2] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models https://arxiv.org/abs/2308.13137

[3] An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs https://arxiv.org/abs/2404.14047v2

[4] Extreme Compression of Large Language Models via Additive Quantization ****https://arxiv.org/abs/2401.06118

[5] QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks https://arxiv.org/abs/2402.04396

[6] QTIP: Quantization with Trellises and Incoherence Processing https://arxiv.org/abs/2406.11235