Addition is All You Need for Energy-efficient Language Models

The primary concern is hardware support, as current GPUs are highly optimized for multiplication operations. If model inference relies solely on summation, a significant portion of the GPU's resources dedicated to multiplication would be underutilized, leading to inefficiency.

1. The paper contains several spelling errors, such as 'Reportly' which should be 'Reportedly,' 'of the save size' which should be 'same size,' 'highlighted in read' which should be 'in red,' as well as 'slighly' for 'slightly,' 'evenlly' for 'evenly,' and 'coparing' for 'comparing,' among others.
2. The selection of L(k) is based on Figure 3, which only includes experimental analysis for the Llama3.1-8b and Gemma2-2b models, making it difficult to establish the general applicability of the current L(k) values across different models.
3. The experimental section lacks model diversity, as it only includes Mistral-7b, Llama3.1-7b, and Gemma2-2b. Additional experiments could be conducted on models such as OPT and Mamba, as well as across different model scales, such as Llama3-13B and Llama3-70B.
4. There is a lack of hardware experiments, with metrics like gate complexity and power consumption only theoretically estimated. Power consumption calculations are based on results published in 2014, which provides limited guidance to the readers. It's suggested that the authors provide RTL implementations, synthesis, and simulation results to demonstrate the ideas.
5. From the perspective of hardware designers, similar approximation techniques have been widely discussed. The authors are required to provide more discussions and comparisons.
6. Some transitions lack clarity. For instance, moving from Equation (1) to Figure 2 requires readers to infer the offset transformation, which could be more explicitly explained.
7. The current experiments cover a variety of tasks, but the scope of applicability for the L-Mul algorithm is not clearly defined.

While the proposed approximation method is interesting, the practical value of simply replacing floating-point multiplications with high-precision additions is not convincing, especially in the context of LLMs.

• First, in an LLM serving system (software + hardware), it is known that the on-chip and off-chip data movement and storage (induced by weights and kv caches), rather than computations, dominate energy consumption. Although computations are streamlined, the proposed method introduces non-negligible overheads on memory footprint and data traffic (due to high-precision additions), which might offset the energy saving of computations.

• Second, I think a system-level comparison with INT4 quantization on a real system (e.g., FPGA-based system) is helpful to demonstrate the efficiency of the proposed method, since the energy consumption of an INT4 MAC is also far less than that of FP, and the storage and bandwidth requirements of INT4 data can also be significantly reduced.

Moreover, the authors claim that the proposed method is orthogonal and complementary to methods that optimize I/O and control. However, evaluations of this (such as incorporation with weight/kv cache pruning) are missing.

• "is all you need" title format is generally overused and in this case does not even fairly represents the content of the paper: most experiments leave regular matmul in place everywhere across the network, with the exception of the attention matmuls. I strongly recommend the authors to use a more informative title for this paper
• a main limitation is that the matmul replacement (and related accuracy results) is primarily focused on attention modules. There is only one very limited example (Table 6) where L-Mul is applied across the full model. What reasons drove the authors to this apply L-Mul only to the attention, in the first place?
• discussion on energy savings is inconsistent and confusing:
  • 95% saving for element-wise computation mentioned in the abstract is not discussed further in the paper
  • 80% saving for dot product is mentioned at the beginning of Section 3.3 but not supported by evidence (it's unclear how this is derived, and for what precision of L-Mul)
  • it is not clear at all how these two numbers are derived, may the authors clarify this?
  • crucially, a discussion of energy saving at model level is not made. I would encourage the authors to provide such estimate
• energy savings are estimates (not measurements) and the behavior of the L-Mul algorithm is simulated, with the authors leaving as future work an actual implementation of this concept at hardware level
• the meaning of Fig 3 should be better explained: is this the combined mse of every matmul and every example of GSM8k, for the two given models? Or a subset of these combinations?
• the abstract mentions that L-Mul with 4 and 3 bits mantissa (= fp10 and fp9, respectively) achieve comparable performance to fp8_e4m3 and fp8_e5m2, respectively. This is reported in Fig. 3 and 4. However, most of the results (Table 2,3,4, possibly 6) relate to L-Mul with 6 bits mantissa (fp12). At least this is my interpretation of the text, the precision used across different experiments is not stated as clearly as it should have been, and I would encourage the authors to amend the text throughout to clarify this point
• the paper shows an embarrassing lack of proofreading: it is full of typos, minor errors, and does not use notation consistently. Some examples include:
  • a sentence is repeated in the abstract with slightly different wording
  • Section 2.1: element-size -> element-wise
  • Section 2.1: A and X matrices are defined to be (N,k) but these dimensions do not match the subsequent calculations:
    • Y1 element-wise product of (N,k) matrices involves N*k multiplications, not N^2
    • Y2 dot product uses N * N * k multiplications, not m * n * k (m was defined as mantissa bits); the number of additions is similar but not exactly the same
  • Section 2.1: a comparison is made between fp16 and int16 energy numbers but the origin of the int16 numbers is not reported (they are not provided in Table 1 along with the other formats)
  • Fig 2 caption: coparing --> comparing
  • Section 2.2 (and elsewhere): the distinction between L-Mul and L-MatMul is unclear and appears to be inconsistent. They are the same op applied to different part of the network. Reported results usually refer to L-Mul but in most cases the new op is applied to the attention matmuls (which eq. 3 calls L-MatMul). Future work talks about L-Mul and L-Matmul. Would recommend to use a single name, as the op is the same
  • Section 2.2: when l(m) is defined in Eq. 1, the text states that it "is defined according to observation in Figure 3" but at this point it is totally unclear what observations are being referred to
  • Section 2.3.1: the meaning of the first sentence is unclear, the grammar is incorrect
  • Section 2.3.2: the texts mentions "the total amount of gate-level computations needed by fp8 Mul" but reports fp16 as well
  • Eq. 4 uses k,r in the explicit form of e^k_mul but f1 is then expressed as function of m,k instead (m being k+r)
  • fp8_m4e3 and fp8_m5e2 baseline errors are used in Fig. 3 as reference but their value is only reported later in Fig. 4!
  • Fig 3 caption: read --> red
  • Section 3.2: "the experimental results further confirm that L-Mul is more efficient and accurate than FP8 multiplication" --> would probably remove this sentence. There aren't additional experiments on efficiency. Accuracy is reported in the following section
  • Appendix A uses n for number of mantissa bits that are called m everywhere else