SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators

The author mentions in Section 4.1, lines 356-358, that to ensure a fair comparison with QuIP# and Omniquant, no fine-tuning was performed on them. This is a fair comparison for QuIP#, which combines codebook and fine-tuning of pre-trained parameters to improve performance. However, Omniquant does not fine-tune any pre-trained parameters, instead using block-wise gradient propagation to update the quantizer parameters, including the scaling factor and zero factor. By not using this technique in the comparison, the author is essentially not using the Omniquant method, but rather a basic statistical quantization. And, AWQ also uses calibration data to pre-compute the scaling parameters. This comparison is unfair and may cause confusion for readers. Additionally, the data in Table 2 is different from what is reported in the AWQ paper, and the author should provide an explanation for this discrepancy.

Thank you for your detailed observations. Regarding AWQ, the numbers reported in the AWQ paper are based on a group size of 128. As explained in our paper, we use 3- or 4-bit integers with channel-wise scaling to ensure that the required bits per element remain within the allocated 3 or 4 bits. A group size of 128, as used in AWQ, introduces an additional quantization overhead of approximately 0.25 bits per parameter. Consequently, the perplexity results for AWQ in Table 2 and the evaluation accuracies in Table 3 are based solely on channel-wise scaling. In this case, the AWQ codebase defaults to using the entire model channel width as the group size, resulting in negligible quantization overhead compared to the 0.25-bit overhead per parameter with a group size of 128. We will clarify this distinction further in the paper. Your point regarding OmniQuant is valid, and we will address it in the paper to ensure readers fully understand the differences in methodology. That said, it is important to highlight that SeedLM remains the only data-free approach in the comparison, achieving competitive or superior results. We appreciate your feedback and will incorporate these clarifications to enhance the paper.

AWQ can quickly determine the scaling of weights and perform quantization through calibration, and the compression time for a 7B model is only a few minutes. However, the LFSR technique proposed in the paper involves matrix decomposition and optimization approximation, and the efficiency of this compression process for extremely large-scale LLMs is lacking in discussion and comparison.

Thank you for raising this important point. In the paper, we highlight that the heuristics are applied across all weight blocks in parallel to identify the seeds and coefficients that minimize reconstruction error, aiming to enhance computational efficiency. To achieve this, we precompute and cache the pseudo-inverse matrices for all seeds and perform the heuristics in parallel across all blocks.

The seed search process is inherently parallelizable, as it can be conducted independently across both seeds and blocks, allowing efficient scaling for large models. While compression time varies depending on configuration parameters such as seed bit-width, block size, and latent dimension, the use of these parallelization strategies enables significant efficiency gains. For reference, we were able to perform all compression tasks, including up to a 70B-parameter model, using a single 80GB A100, with runtimes ranging from minutes to hours depending on the model size and configurations. We will elaborate on these details further in the paper.

Thank you for your thoughtful and detailed feedback. We greatly appreciate your recognition of the innovative aspects of SeedLM, especially its data-free approach and the use of pseudo-random generator seeds for efficient weight compression. We're pleased to hear that you found the FPGA implementation and its potential applications in hardware-constrained environments valuable. Eliminating the need for calibration data would indeed simplify the compression process, reduce quantization offset issues, and can maintain high accuracy while achieving good compression ratios.

We appreciate your feedback and will carefully address the areas you highlighted to further improve the work. Below, we provide our responses to your specific points:

The author compared the AWQ, Omniquant, and QuIP# methods. However, Omniquant and QuIP# were primarily designed for ultra-low bit-width quantization compression, such as 2-bit, but the author only compared the performance of 3/4-bit and did not show the quantization results of 2-bit. In the field of LLM quantization, SOTA methods specifically designed for 4/3-bit, such as GPTQ[1], were not included in the comparison. This makes the results unconvincing.

Thank you for pointing this out. SeedLM primarily focuses on 3- and 4-bit configurations, and we will make this clearer in the paper by adding the following to the discussion:

"SeedLM primarily focused on 3-bit and 4-bit compression, applying the same design choices across all blocks for these levels. However, we believe that pushing below the 3-bit level might require different configurations tailored to each block to manage compression noise effectively or leveraging data for additional fine-tuning to enhance results. We leave these directions for future work."

We appreciate you highlighting GPTQ as a strong method for 3/4-bit quantization. Initially, we did not include it in our comparisons because the baselines we selected were reported in the literature as superior or comparable to GPTQ. However, we recognize the importance of including GPTQ in our comparisons. Below, please find the perplexity results obtained using the same WikiText-2 setting as in Table 2 for 4-bit quantization. These results show that SeedLM consistently outperforms GPTQ in 4-bit configurations. For 3-bit quantization, GPTQ results appear significantly lower; for instance, for LLama2-13B in 3-bit, GPTQ yields a perplexity of 9.5 compared to SeedLM's 5.8. We will include these findings in the ablation section of the paper's appendix for completeness.

The overall writing of the article is not clear in some details. In Eq (1), I can infer that the compression matrices for LFSR are U and t, but the details of how the input activation X is computed with the LFSR-compressed weights during the actual inference decoding stage are not discussed in detail. The article should provide more information on how the LFSR-compressed weights are used in the inference process and how they differ from other quantization methods. This would help to clarify the advantages and characteristics of the LFSR-based method.

Thank you for your feedback. This approach differs from traditional quantization methods by leveraging a pseudo-random generator to dynamically reconstruct weights during inference. This design specifically targets speeding up memory-bound tasks, as highlighted in the paper. In response to your feedback, we will add the following explanation to the paper before Equation 1 to clarify the inference process:

"During inference, $s$ and $\mathbf{t}$, which require fewer bits when considering all the bits in a block compared to the original weights, are retrieved from memory. This reduces the memory footprint and accelerates memory-bound generation tasks. The weights are reconstructed on-the-fly using $s$ and $\mathbf{t}$, and these reconstructed weights are then used to compute intermediate activations and, ultimately, the outputs."

We appreciate your valuable feedback and hope our responses have addressed your concerns, warranting a re-evaluation of the score. We welcome further discussion of any remaining questions during the discussion period. Thank you for your time and consideration.

Thank you for your thoughtful and detailed feedback. We’re delighted that you appreciated SeedLM’s approach, especially its innovative use of arbitrary data formats with shared exponents and its focus on a hardware-friendly FPGA implementation. We’re pleased to see recognition of SeedLM’s data-free nature, which distinguishes it from methods relying on calibration data or fine-tuning. Additionally, we appreciate your acknowledgment of the memory efficiency and speedup advantages. We deeply value your feedback and will carefully address the areas you highlighted to further enhance the work. Below, we provide detailed responses to your specific points:

Weakness 1: Absence of GPTQ Comparison: The paper does not provide a comparison with GPTQ, a commonly used quantization baseline, which is a notable omission given GPTQ's relevance to LLM compression.

Question 1: Comparison with GPTQ : Why was GPTQ omitted from the comparisons? Can you clarify its potential impact on results, given that it’s a standard benchmark in LLM compression?

We acknowledge GPTQ as a strong compression method, and the reason we initially excluded it from our comparisons was that the baselines we selected were reported in the literature as either superior or comparable to GPTQ. However, we recognize the importance of including GPTQ in our evaluations. Below, please find the perplexity results obtained using the same WikiText-2 setting as in Table 2 for 4-bit quantization. These results show that SeedLM consistently outperforms GPTQ in 4-bit configurations. For 3-bit quantization, GPTQ results appear significantly lower; for instance, for LLama2-13B in 3-bit, GPTQ yields a perplexity of 9.5 compared to SeedLM's 5.8. We will include these findings in the ablation section of the paper's appendix for completeness.

Weakness 2: Inference Efficiency Assumptions: While the paper mentions using the latest repositories, many of these codebases likely store compressed weights in full precision during inference, leading to potential memory inefficiencies.

Question 3: Full Precision Storage Impact: Have the authors tested SeedLM with full precision storage during inference? If so, could this be the cause of out-of-memory issues observed with some baseline methods?

Thank you for your observation. SeedLM is specifically designed to avoid storing weights in full precision during inference by using a custom decoding mechanism that reconstructs weights on-the-fly during inference. This approach minimizes memory usage, speeding up memory-bound tasks without the need for full-precision storage. The out-of-memory issue observed with Quip# is due to the potential inefficiencies in the Quip# repo. We’re working with the Quip# team to address this issue and will ensure the manuscript is updated accordingly.

Weakness 3: GPU Implementation Challenges: Although FPGA implementation is shown, the challenges of porting SeedLM to GPUs are unaddressed. Issues like increased kernel launches for memory-bound tasks and limited support for LFSRs in recent GPU hardware could impact performance.

Question 4: GPU Implementation Challenges: Could the authors comment on the challenges of implementing SeedLM on GPUs, specifically the support for LFSR in recent GPUs and whether additional kernel launches could reduce performance for memory-bound applications on NVIDIA hardware?

Thank you for highlighting this point. Implementing SeedLM on GPUs does present challenges, such as increased kernel launches for memory-bound tasks and the limited support for LFSRs in current GPU hardware. These factors can affect performance, as GPUs are less optimized for the on-the-fly weight reconstruction approach used in SeedLM. Additionally, the data type for the latent vector may not be natively supported, adding to the complexity. That said, it’s important to note that hardware capabilities—across various platforms, not just GPUs—are continually evolving. LFSR is a particularly hardware-friendly method, well-suited to low-area, low-energy, and high-throughput implementations. One of the key aims of this paper is to demonstrate the practicality of SeedLM’s approach, including LFSR-based techniques, and to provide evidence that could motivate the development of new hardware features to better support such methods. While we focused on FPGA for its current advantages, we will add the following discussion to the paper to address broader possibilities and future hardware compatibility:

“as hardware continues to evolve, SeedLM's applicability could expand, with its LFSR-based design offering low-area, low-energy, and high-throughput benefits. This work highlights the practicality of such techniques, aiming to inspire hardware innovations that support broader device compatibility, including GPUs.”

Weakness 4: Optimization Overhead for Parameter Selection: The process of determining optimal parameters for each weight block, such as the seeds, coefficients, and latent dimensions, may introduce significant overhead.

Question 5: Parameter Optimization Overhead: What is the computational cost of finding optimal seeds and coefficients? Could the process be streamlined for faster deployment?

Thank you for raising this important point. In the paper, we highlight that the heuristics are applied across all weight blocks in parallel to identify the seeds and coefficients that minimize reconstruction error, aiming to enhance computational efficiency. To achieve this, we precompute and cache the pseudo-inverse matrices for all seeds and perform the heuristics in parallel across all blocks.

The seed search process is inherently parallelizable, as it can be conducted independently across both seeds and blocks, allowing efficient scaling for large models. While compression time varies depending on configuration parameters such as seed bit-width, block size, and latent dimension, the use of these parallelization strategies enables significant efficiency gains. For reference, we were able to perform all compression tasks, including up to a 70B-parameter model, using a single 80GB A100, with runtimes ranging from minutes to hours depending on the model size and configurations. We will elaborate on these details further in the paper.

Question 2: Compression Limitations for LLaMA 3: Many methods struggle to compress LLaMA 3 effectively. Do the authors have insights on why this model, in particular, is challenging to compress?

Question 6: Sensitivity in Compressing LLaMA 3: Many quantization techniques, appear to struggle with LLaMA 3, indicating potential limitations in compressing this model type. What features does this family of models have that makes it difficult for other methods to quantize, and why does SeedLM perform well in this case?

Thank you for this insightful observation. Compressing LLaMA 3 indeed presents unique challenges, likely due to its extensive training on a vast number of tokens. As mentioned in the paper, this is probably a result of its advanced architecture and significantly larger training dataset. The LLaMA 3 introduction in Meta’s blog states:

“We made several new observations on scaling behavior during the development of Llama 3. For example, while the Chinchilla-optimal amount of training compute for an 8B parameter model corresponds to ~200B tokens, we found that model performance continues to improve even after the model is trained on two orders of magnitude more data. Both our 8B and 70B parameter models continued to improve log-linearly after we trained them on up to 15T tokens.”

Models trained on such large datasets capture intricate patterns, which may make them more sensitive to compression techniques. This aligns with insights from the recent ICLR submission "Scaling Laws for Precision," which just came to our attention. The study observes that degradation from post-training quantization increases with the volume of training data, making models like LLaMA 3 more susceptible to performance loss during compression. We will incorporate this insight and citation into our paper to further clarify the compression challenges faced by highly trained models. As highlighted in Appendix G of this study, the authors speculate that this sensitivity may be arising from sharpness and hierarchical learning strategies developed during training. This suggests that LLaMA 3's sensitivity to compression may stem from its reliance on deeply hierarchical and sharply tuned features, making traditional data-reliant compression methods less effective. Data-dependent approaches likely require a substantial number of tokens to adapt effectively to such highly trained models, potentially defeating the purpose of efficient post-training compression. In this context, we speculate that SeedLM incurs a lower initial compression error when cast into its compressed form, keeping it closer to the model's final pretrained state. Its data-free approach appears particularly well-suited, as it can eliminate the need for additional adaptation data. SeedLM’s ability to succeed without such data highlights its robustness in handling the intricate structure of these advanced models.

Thank you once again for your thoughtful feedback. We hope our responses have effectively addressed your concerns and merit a re-evaluation of the score. We welcome further discussion and are happy to address any additional questions during the discussion period. We appreciate your time and consideration.

Dear Authors,

I appreciate the detailed discussion to address my questions and concerns and look forward to seeing these clarifications and improvements incorporated into a revised version or the camera-ready manuscript. Additionally, I am eager to see how you address the memory-related challenges in comparison to related work. I maintain my positive assessment of your submission with a score of 6.

Best regards,

Reviewer WtPj

Thank you for your thoughtful and detailed feedback. We’re pleased that you found the novelty of our method, particularly the use of a pseudo-random generator for efficient code storage and retrieval, to be valuable. We’re also glad that the quality of our presentation, including the detailed formulas, diagrams, and the implementation focus in the Performance Analysis section, resonated with you. We are actively working to address the comments you raised, which we believe will further strengthen the paper. Below, we provide detailed responses to your specific points:

Weakness 1: Comparison with other quantization methods is incomplete. Most striking shortcoming is lack of comparison with finetuned model which is what most current SOTA models use.

Question 1: Please compare your method with SOTA methods which use finetuning after quantization.

Thank you for raising this insightful point. While fine-tuning after quantization certainly has its merits, our work offers a complementary perspective with significant potential. SeedLM achieves state-of-the-art results in 3/4-bit quantization without requiring any data. Additionally, beyond the simplicity of a data-free approach, it may help mitigate compression offset issues, especially when the distribution of fine-tuning data is not plausible. That said, we acknowledge the importance of exploring fine-tuning strategies on top of SeedLM, which we have highlighted in the concluding remarks as a future direction. We are actively investigating effective fine-tuning strategies for SeedLM and agree with you that it has the potential to push the bit-rate even further down while maintaining near iso-accurate performance. While we couldn’t conclude these experiments in the rebuttal period, we will ensure that our findings are included in the final version of the paper, should it be accepted.

Weakness 2: The paper dismisses comparison with strong methods like AQ, SPQR in desire to "avoid costly training". Yet these are quite good benchmarks to compare with, theu=y have reported figures, and to larg share of practitioners the extra training time (hours actually) could be acceptable.

Question 7: Please provide comparison to AQ, SPQR, as these are strong and valid SOTA benchmarks

Thank you for highlighting this point. These are indeed excellent benchmarks, and we agree that they represent strong and valid SOTA comparisons. We are actively working to obtain those numbers and will include them in the paper. Thank you for bringing these up.

Weakness 3: some of the results in table 2 are not consistent with those published in the respective papers. for instance, AWQ claims 3.41 perplexity for L2-70, (https://arxiv.org/pdf/2306.00978, table 4) while you indicate 3.5 (Table 2)

Thank you for your detailed observations. Regarding AWQ, the numbers reported in the AWQ paper are based on a group size of 128. As explained in our paper, we use 3- or 4-bit integers with channel-wise scaling to ensure that the required bits per element remain within the allocated 3 or 4 bits. A group size of 128, as used in AWQ, introduces an additional quantization overhead of approximately 0.25 bits per parameter. Consequently, the perplexity results for AWQ in Table 2 and the evaluation accuracies in Table 3 are based solely on channel-wise scaling. In this case, the AWQ codebase defaults to using the entire model channel width as the group size, resulting in negligible quantization overhead compared to the 0.25-bit overhead per parameter with a group size of 128. We will clarify this distinction further in the paper.

Question 2: What is your method performance when quantizing into 1.5-2.5 bits per weight? This area is where most quantization research progress is happening, your method may be competitive there too.

Thank you for pointing this out. SeedLM primarily focuses on 3- and 4-bit configurations, as going below 3 bits without data may require custom adjustments (e.g., seed bit-width, block size, latent dimension, and latent data type) to better address compression noise based on the weight values. Alternatively, it could involve leveraging data and applying fine-tuning on the models, adapters, or similar adjustments. We believe that achieving competitive results with SeedLM below 3 bits starts to depend on such techniques, and we will elaborate on these points in the paper as discussed.

Question 3: please address the OOM situation in Quip# L2-70 benchmarks (Table2). What is different in your setup vs the original one from https://arxiv.org/pdf/2307.13304? What have you tried to avoid OOM?

Thank you for your observation. The out-of-memory issue observed with Quip# is due to the potential inefficiencies in the Quip# repo. We’re working with the Quip# team to address this issue and will ensure the manuscript is updated accordingly.

Question 4: Add performance timing benchmarks vs other methods, this solidifies your claim of speed advantage.

Thank you for this suggestion. Each method we evaluated requires specific hardware adjustments to optimize performance, which makes direct comparisons challenging. In future work, we’ll explore ways to standardize hardware setups to facilitate clearer comparisons. For now, we chose bit width as a standardized metric because it is widely used in the literature and reflects the weight's average bits per element, which directly influences memory access and correlates with speedups in memory-bound tasks.

Question 5: How would speedup of the method change after post-quantization finetuning?

Thank you for the question. Post-quantization finetuning would have a minimal impact on inference latency, as runtime is not heavily influenced by the specific learned weights. SeedLM’s speed advantage primarily arises from the efficient use of pseudo-randomly generated weights and reduced memory access, which remain consistent regardless of whether finetuning is applied.

Question 6: Unlike some other submission, this one has no implementation code. I wonder if a private code repository could be provided for review purposes?

Unfortunately, due to our institution's policy, we’re unable to comply with the anonymized requirements outlined in the ICLR guidelines. We recognize the importance of reviewing both the code and the reported findings, and we’re committed to working with our institution to find ways to share the code publicly once the review is complete.

Thank you for your valuable feedback. We hope our responses have addressed your concerns and will incorporate the changes in the paper as discussed. If you have any remaining questions, we’re happy to discuss them further during the review period. We appreciate your time and consideration.

Dear Reviewer,

Thank you for your thoughtful comments and for providing a helpful list of current SOTA methods to consider. We are working to include comparisons with some of these methods, as you suggested. We will also ensure that the camera-ready version includes a detailed discussion of the GPU implementation challenges, as requested. We understand the importance of this aspect for industry adoption and will address it accordingly.

Thank you again for your constructive feedback and positive assessment of our work. We truly value having had the opportunity to engage with you during this discussion period.