ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization

We greatly appreciate your careful review and constructive suggestions. Below are our detailed responses to your concerns.

W1: The multi-objective optimization method is quite novel and impressive, but it needs to be tested with more cases.

Thank you for acknowledging the novelty of our multi-objective optimization method! Following your suggestions, we’ve conducted additional tests and comparisons on the OPT and LLaMA model families.

First, we conducted additional ablation studies using both OPT and LLaMA models, supplementing the results in Table 8 of the submitted manuscript. The new results are presented in Table 5 of the attached PDF in our global response. Our multi-objective optimization demonstrates superior performance, achieving average perplexity reductions of 20.85/8.56/112.26 on OPT models and 1.00/1.21/2.25 on LLaMA models compared to weight-only objective, activation-only objective, or their vanilla combinations, respectively.

Second, we tested the average layer-wise quantization errors for a direct comparison. The results are shown in the table below. Our multi-objective optimization achieves significantly lower quantization errors for both weights and output activations. Specifically, we observed average per-parameter weight quantization error reductions of 0.4 and 0.1, and total output activation error reductions of 18.9 and 53.1 compared to OPTQ and LUT-GEMM, respectively.

These tests demonstrate that our multi-objective optimization achieves lower quantization errors and better model accuracy compared to previous methods focused solely on weight-only or activation-only objectives.

---

W2: The use of mixed precision for layers with different sensitivities is a good idea, but it is designed for one LLM model. How effective is the automated bit allocation? And how applicable is this approach to different LLM models?

Thank you for your constructive questions! We agree that the effectiveness and applicability of our automated bit allocation need to be validated across different LLM models.

To address your questions, we evaluated our mixed bit allocation strategy and compared Ours (Mixed) with Ours (Lat.). The results are shown in Table 6 of the attached PDF in our global response. Ours (Mixed) further reduces perplexity by an average of 96.86, 3.23, and 2.63 for OPT, LLaMA, and Gemma models, respectively, under comparable or even less latency. This set of experiments further validates the applicability of our automated bit allocation strategy to different LLMs.

In addition, we want to clarify that, for each model, we search for the optimal bit allocation with negligible overhead (e.g., 1%~10% of the reparameterization time). For example, it takes 0.5 seconds for searching versus 72 seconds for reparameterizing OPT-125M with a single bit configuration, and 1 minute for searching versus 13 minutes for reparameterizing OPT-13B with a single bit configuration. This is achieved by leveraging the proposed proxy criteria (as shown in Eq. 3 of the submitted manuscript), instead of searching according to the reparameterization errors, which is time-consuming and requires running models at each bit. Using the proxy criteria, the bit allocation candidate rankings are highly correlated with the rankings obtained using actual reparameterization errors, with a Kendall $\tau$ of 0.910/0.905/0.915 for OPT-125M/1.3B/13B and 0.931/0.929/0.897 for LLaMA-7B/13B/8B, respectively.

In summary, our proposed automated bit allocation strategy is effective and applicable to different LLMs with negligible overhead using the proposed proxy criteria.

We appreciate your time and suggestions in reviewing our work. Below are our detailed responses to your concerns.

W1 & L1: Discuss the impact of batch sizes on throughput. If only a batch size of 1 is considered, explain why this is practical?

Following your suggestion, we have further tested the throughput of our CUDA kernels and end-to-end models with increased batch sizes, as demonstrated in Figure 1 of the attached PDF in our global response. Our ShiftAddLLM still outperforms all three baselines at a batch size of 8 in terms of accuracy-efficiency trade-offs, achieving on average 3.37x/2.55x/1.39x throughput improvements compared to OPTQ, AWQ, and LUT-GEMM at similar or much better accuracy.

Previously, we assumed a batch size of one for mobile applications where only one user is using the LLM. This assumption also stems from the sequential nature of LLMs during generation, i.e., generating one token at a time based on all previously generated contexts. The assumption of a batch size of 1 is also used in previous literature, such as AWQ, OPTQ, and LUT-GEMM, to measure the latency or throughput for LLM serving. We will clarify this assumption.

W2: The authors show the benefits via an Eyeriss-like H.W. simulation, but GPUs already have numerous multiplication units. Rewritten to address the new ASIC design if that is focus?

We humbly clarify and emphasize that we reported real-measured GPU latency (see Lines 281-282 and Figure 6 of the submitted manuscript) instead of simulated results to demonstrate the up to 65% latency savings, which benefit from our proposed shift-and-add reparameterization and dedicated CUDA kernel optimization for GPUs.

Regarding energy savings, since we cannot directly measure energy on GPUs, we used existing simulators to estimate the energy savings. However, our focus remains on showing the practical benefits of current GPUs. We are not aiming to propose a new ASIC design in this work, as that is not our main focus.

W3: Benchmark with FlexRound and OmniQuant?

As suggested, we further compare our ShiftAddLLM with both FlexRound and OmniQuant on OPT and LLaMA models. As shown in Tables 2 & 3 of the attached PDF in our global response, our ShiftAddLLM consistently shows better accuracy-efficiency trade-offs, achieving average 0.15 (4-bit) / 0.39 (3-bit) and 0.30 (4-bit) / 0.52 (3-bit) perplexity reduction, as compared to FlexRound and OmniQuant, respectively. Note that the baseline results are directly obtained from the original paper and follow-up work LRQ [1]. In addition, we tested OmniQuant at 2 bits ourselves and found it fails for OPT models, whereas ours performs well for OPT models and also achieves an average 1.96 perplexity reduction than OmniQuant on LLaMA at 2 bits.

[1] LRQ: Optimizing PTQ for LLMs by Learning Low-Rank Weight-Scaling Matrices, arXiv'24

W4: Skeptical about using perplexity as the main metric. Show scores for MMLU and conduct A/B tests?

We acknowledge that perplexity is not the gold metric and therefore also provided the accuracy of seven downstream tasks in Table 5 of the submitted manuscript.

Furthermore, as suggested, we extend the evaluation to include the MMLU and A/B test using AlpacaEval with GPT-4. The results are shown in Table 4 and Figure 2 of the attached PDF in our global response. Our ShiftAddLLM consistently achieves 3.58% accuracy improvements over OPTQ and 3.83% over LUT-GEMM for MMLU when using the OPT-60B model. For A/B tests, we used the GPT-4 score to evaluate the quantized models against the FP16 counterpart. Our ShiftAddLLM achieves 8.6%/20.7% higher winning rates than OPTQ (29.3%) and LUT-GEMM (17.2%) when using the LLaMA-2-7B model. We provide an example of the generation comparison in Figure 2 of the attached PDF.

W5 & L2: Do not present results for a 4-bit setup. Why select such extremely low-bit methods like 3/2 bits only?

As requested, we have provided the 4-bit results in Table 1 of the attached PDF in our global response. These results show that ShiftAddLLM consistently outperforms the baselines at 4 bits, achieving average perplexity reductions of 0.90/1.32/1.00 and 0.44/0.22/0.02 as compared to OPTQ / LUT-GEMM / AWQ, using OPT models and LLaMA models, respectively. In addition, we have also included comparisons with FlexRound and OmniQuant at 4 bits (see response to W3).

We consider lower-bit quantization because we aim to push the accuracy-efficiency boundary to lower bits with minimal accuracy compromise. This is meaningful for large-scale LLMs, where even at 3 bits, they remain memory-bound. As analyzed using the Roofline model shown in Figure 5 of [2], for Nvidia A6000 GPUs, the turning point from memory-bound to compute-bound is 200 arithmetic intensity (OPs/bytes). For LLaMA-7B models, all the operators in the decode/generation phase have around or less than 1 arithmetic intensity, as shown in Table 1 of [2]. Even at 4 bits, the arithmetic intensity is approximately 1 / 3 * 32 = 8 (same ops but 4/32 fewer memory accesses), which is far less than the turning point of 200 and thus remains memory-bound, let alone larger models like LLaMA-70B or beyond. Reducing from 4 bits to 2 bits can help increase the arithmetic intensity and thus the theoretically maximum performance by 2x, from 6144G OPS to 12288G OPS. If memory is not a bottleneck for much smaller cases or prefill stages, higher bits can be used for better accuracy. Our goal is to offer an additional option and trade-off for large, memory-bound cases, without forcing the exclusive use of 2 bits.

[2] LLM Inference Unveiled: Survey and Roofline Model Insights, arXiv'24

L3: Difficult to estimate latency improvement. Plan to release the code?

Yes, we do plan to open-source the code to ensure reproducibility, as we also promised in the abstract.

We have reported the real-measured GPU latency. Our ShiftAddLLM achieves 6.5% ~ 65.0% latency reductions for OPT/LLaMA models, at similar or even lower perplexity.

Dear Reviewer,

Could you please review the rebuttal, discuss it with the peer reviewers, and finalize your rating?

Thank you for your efforts!

Regards,

AC

Response [1/2]

---

Dear Reviewer 7Vce,

​We were very encouraged by your previous feedback that our initial rebuttal successfully addressed your major concern regarding batching results and clarified both the motivation for very low-bit quantization and our real-GPU latency (as opposed to using a new ASIC). We believe that adding these additional experimental results and clarifications will further strengthen our work and its contributions to the community. Regarding the new discussion, we humbly seek to clarify our points and hope to reach a consensus in our understanding.

For the first point, there were potential misunderstandings. We agree with you that MSFP shares exponents and shifts the mantissa accordingly, mimicking multiplication by powers of two. However, while we recognize MSFP’s unique contributions/innovations well, we humbly clarify that our approach differs from MSFP in two key aspects:

1. Nature of Approach: MSFP uses shared exponents but relies on various shifted mantissa to represent the weights; without this, all weights would collapse to the same value. In contrast, we do not use shared exponents for scaling factors and eliminate the need for mantissa. In particular, each scaling factor is represented as a distinct power-of-two integer (equivalent to the exponents in floating-point numbers, completely removing the mantissa bits). In this way, the multiplication between a floating-point activation and a power-of-two integer scaling factor can be simplified to adding the corresponding integer to the exponent bit of the floating-point activation, as described in Figure 1(c) of the submitted manuscript. In addition, rather than sharing the exponents, the entire scaling factor in ShiftAddLLM is shared across groups of binary weights in a column/block-wise manner, as illustrated in Figure 3(a) and detailed in Section 4.2 of the submitted manuscript, carefully designed to optimize both weight quantization and output activation errors without conflicts. Hence, there are clear differences between the MSFP datatype and our quantization scheme. In fact, our method is orthogonal to MSFP and can be combined with it by representing input activations in MSFP for more aggressive performance improvements.

2. Determining Shared Exponents or Scaling Factors: The method for determining shared exponents in MSFP or shared scaling factors in our quantization scheme is different. MSFP selects the maximum exponent to share across the bounding-box size, i.e., the number of elements sharing one exponent [1], which is simpler in implementation yet might not be as adaptive. In contrast, in our ShiftAddLLM, the reparameterized binary weights and scaling factors result from multi-objective optimization. This optimization adaptively designs scaling factor patterns to avoid conflicts between optimizing weight errors and optimizing output activation errors.

Finally, in terms of the performance outcomes, MSFP at 4 bits (1-bit sign and 3-bit mantissa) already suffers from large quantization errors, as evidenced by the significant KL divergence shown in Figure 3 of [1]. In contrast, our ShiftAddLLM at 3 or 4 bits can still achieve comparable accuracy to FP baselines.

To directly compare ShiftAddLLM with MSFP, we conducted additional experiments to compare (1) quantization errors and (2) KL divergence using both methods against their floating-point counterparts. We randomly selected ten weight matrices from OPT-350M, quantizing or reparameterizing them using both methods. The results, summarized in the tables below, indicate that ShiftAddLLM consistently outperforms MSFP, achieving lower KL divergence by 0.0065, 0.0271, and 0.0952, and reducing quantization errors by 1707.3, 3251.1, and 5862.0 at 4-bit, 3-bit, and 2-bit quantization, respectively.

[1] Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point, NeurIPS’20

Response [2/2]

---

For the second point, as emphasized in our submitted manuscript, our primary focus is on GPU acceleration, specifically through the development of dedicated CUDA kernel support. It is worth noting that, we intentionally did not delve into specific ASIC designs, which were referenced only to demonstrate potential energy savings. We hope to humbly clarify that while we appreciate and follow your suggestion to describe the ASIC design used in our energy saving experiment, it is not the primary target of our work. In addition, our ShiftAddLLM has been evaluated and outperforms the baselines on both (1) a single GPU (bs=1) for single-user interactions, which is to follow the settings in SOTA prior works like AWQ, OPTQ, and LUT-GEMM and can be used for mobile applications, and (2) cloud data-center setups (bs > 1) supporting multiple users.

Finally, as recognized by the other two reviewers, our method's contributions extend beyond reparameterization. They also include (1) the crucial multi-objective optimization and (2) the automated mixed-bit allocation strategy, for which we have provided additional results to fully demonstrate their effectiveness (please refer to our response to Reviewer 4dkb’s W1/W2 for more details).

---

In summary, we have provided experiments and clarifications to address all the concerns and comments you raised:

• The impact of batch sizes on throughput and why consider batch size of 1:
  • We have clarified in our reply to your W1 & L1 and included the results in Figure 1 of the attached PDF.
• ASIC Design vs. GPU Acceleration:
  • We have clarified that our results are real-measured GPU speedups.
• Comparison with FlexRound and OmniQuant:
  • We have included the results in Tables 2 and 3 of the attached PDF.
• Show scores for MMLU and conduct A/B tests using AplacaEval with GPT-4:
  • We have included the results in Table 4 and Figure 2 of the attached PDF.
• Lack of results for a 4-bit setup:
  • We have included the results in Table 1 of the attached PDF.
• Why reduce from 4 bits to 3 bits or even 2 bits:
  • We have clarified in our reply to your W5 & L2.
• Comparison with MSFP (asked during the discussion phase):
  • We have clarified the differences and included the comparison in this response.

---

Best regards,

Paper 12025 Authors

We greatly appreciate your positive comments and constructive suggestions. Below are our detailed responses to your questions.

W1: The clarity of the presentation could be improved.

(1) Technique applicability beyond LLMs: discussion needed. You are correct that the idea is general and can be extended to other smaller models like CNNs [1] or ViTs [2]. Meanwhile, this work’s implementation is specifically dedicated to large-scale LLMs: It is the first instance of applying the shift-and-add technique at the scale of LLMs with billions of parameters. While many ideas perform well with models having millions of parameters, they often fail to scale effectively. Unlike previous methods that require additional training and do not yield good results for large-scale LLMs, our approach is uniquely tailored for LLMs. We incorporate "post-training" reparameterization and carefully designed scaling factor patterns, enabling multi-objective optimization for LLMs and ensuring superior performance compared to prior quantization methods. We will add this discussion to the final revision.

[1] ShiftAddNet: A Hardware-Inspired Deep Network, NeurIPS'20

[2] ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient ViTs, NeurIPS'23

(2) Lines 41-42: Refer to the exact numbers. Thanks for pointing this out! The number is derived by comparing adds to multiplications in terms of the INT32 format. The energy savings are 3.1 / 0.1 = 31x, and the area savings are 3495 / 137 ≈ 26x. We will clarify this in the tables.

(3) Lines 83-84: Perplexity reduction or improvements? We apologize for the misuse of the word "improvements." We are actually referring to perplexity reductions, as lower perplexity denotes better results. We will correct this.

(4) Lines 141-146: The presentation of energy savings could be improved. We greatly value this suggestion and consider a summed energy comparison between equivalent computations. We tested matrix multiplication from one MLP layer of OPT-66B between weight $W \in \mathbb{R}^{9216 \times 36884}$ and activation $A \in \mathbb{R}^{1 \times 9216}$ using: (1) FP16 MACs, (2) OPTQ with 3 bits weights and (3) our ShiftAddLLM with 3 bits weights. The resulting energy consumption are 80.36J, 18.48J, and 9.77J, respectively. Our method achieves energy savings of 87.8% compared to FP16 and 47.1% compared to OPTQ.

(5) Lines 218-219: Refer to numbers in the table. The number is derived by comparing the first two rows of Figure 3 (b). The perplexity is reduced by 16.3 - 9.6 = 6.7, and the latency overhead is (44.1 - 33.2) / 44.1 ≈ 24.7%. We will make this clear.

(6) Line 241: Briefly introduce abbreviations. Sure, in self-attention, Q/K/V refers to linear layers for queries, keys, and values, respectively. Out, on the other hand, refers to the output linear layer. In MLPs, FC1 and FC2 refer to the two adopted linear layers. We will clarify this.

(7) Table 3: Discussion on why reducing from 3 bits to 2 bits. Great point! We will add the following discussion: We aim to push the accuracy-efficiency boundary to lower bits with minimal accuracy compromise. This is meaningful for large-scale LLMs, where even at 3 bits, they remain memory-bound. As analyzed using the Roofline model shown in Figure 5 of [3], for Nvidia A6000 GPUs, the turning point from memory-bound to compute-bound is 200 arithmetic intensity (OPs/bytes). For LLaMA-7B models, all the operators in the decode/generation phase have around or less than 1 arithmetic intensity, as shown in Table 1 of [3]. Even at 3 bits, the arithmetic intensity is approximately 1 / 3 * 32 ≈ 10 (same ops but 3/32 fewer memory accesses), which is far less than the turning point of 200 and thus remains memory-bound, let alone larger models like LLaMA-70B or beyond. Reducing from 3 bits to 2 bits can help increase the arithmetic intensity and thus the theoretically maximum performance by 1.5x, from 8192G OPS to 12288G OPS. If memory is not a bottleneck for much smaller cases or prefill stages, higher bits can be used for better accuracy. Our goal is to offer an additional option and trade-off for large, memory-bound cases, without forcing the exclusive use of 2 bits. We will add this discussion to the final revision.

[3] LLM Inference Unveiled: Survey and Roofline Model Insights, arXiv'24

(8) Table 5 lacks an FP16 baseline. We provide the FP16 baseline results in Table 4 of the attached PDF in our global response. Our ShiftAddLLM at 3 bits achieves comparable or even better accuracy than the FP16 baseline, for example, 72.45 vs. 69.82 for BoolQ and on average 0.52% accuracy gain across eight tasks using the OPT-66B model.

(9) Figure 8: Remove the patterns. We will remove the patterns and use the recommended colors.

(10) Remove “Insights” sections. Will remove.

(11) Change the order of tables/figures. Will change orders accordingly.

(12) Correct Section 5.4 title. Will correct the title.

(13) Typos. Will correct it.

---

Q1: Figure 1 could benefit from improved explanations. Intuitively, the binary weights select from the LUTs? LUT output is FP16?

Your intuition is correct. We use binary weights to serve as the key for the LUTs. The "8-bit key" refers to grouped binary weights, with eight binary weights forming an INT8 key. To construct the LUTs, we precompute 256 (2^8) possible values for every eight elements in the shifted activations. Suppose the shifted activation is an n-dimensional vector. In that case, we will get n/8 LUTs, where the grouped binary weights are used as keys, and the precomputed partial sums are stored as values. Yes, the LUT output is in FP16 format. We will add these details to the final revision for clarity.

Q2: Clarify whether using Eyeriss or DNN-chip Predictor.

We used the cited DNN-chip predictor to simulate and calculate the energy (within 18% of the differences with Eyeriss’s chip measurement results as claimed). We will clarify this.

Dear Reviewer 3fe7,

Thank you very much for taking the time to check our rebuttal, providing your positive feedback, and considering raising your score. We are very encouraged to hear that our rebuttal has addressed your comments well. Regarding the batching aspect, we would like to provide further experiment results and clarification as follows,

---

1. Sensitivity Analysis with Larger Batch Sizes.

In line with Reviewer 7Vce’s suggestions and to address the importance of batching in the deployment of LLMs, we have conducted additional experiments focusing on the throughput of our CUDA kernels and end-to-end models with increased batch sizes, from 1 to 8.

As shown in Figure 1 of the rebuttal PDF in our global response, our ShiftAddLLM continues to outperform all three baselines—OPTQ, AWQ, and LUT-GEMM—at a batch size of 8. Specifically, our method achieves throughput improvements of 3.37$\times$, 2.55$\times$, and 1.39$\times$ at iso-quality, respectively, while maintaining similar or even better accuracy (see Figure 1a and Figure 1b in the rebuttal PDF).

---

2. Why Batch Size = 1?

Our initial focus on a batch size of one was guided by the prevalent scenario in mobile applications, where individual user interactions typically involve sequential token generation. This assumption, which has been adopted in prior works like AWQ, OPTQ, and LUT-GEMM, reflects the real-world latency and throughput concerns in LLM serving. But we agree that the cloud is also an important use case, and we will make sure to discuss this case and provide ablation studies in the revised manuscript.

---

We hope these new results and clarification can effectively address your concerns regarding throughput at larger batch sizes. We greatly appreciate that your constructive review has helped further improve and strengthen our work, making our research more valuable for the community.

Best regards,

Paper 12025 Authors