Towards Accurate and Efficient Sub-8-Bit Integer Training

1. Limited performance improvement:

ShiftQuant uses coarser per-group quantization, capping its performance below per-group quantization, and the speed-up gains (e.g., 1.85× on CPU) are not very impressive, experiments lack speed comparison with PSQ.

2. Unclear figures and tables:

Some figures (fig3) and tables(table2,table3) are not well-explained, making it hard to interpret the results clearly.

1. In Section 1: “Our prototypical implementation of ShiftQuant achieves more than 1:85 × /15:3% performance improvement on CPU (ARMv8)/GPU (Nvidia RTX 3090) compared to Pytorch.fp16, and more than 33:9% resource consumption reduction on FPGA (ZC706) compared to FP16.” The comparison with FP16 is not very significant since FP16 is obviously more resource-hungry. It is recommended to compare the proposed method with other existing quantized approaches. Please include comparisons against the current best-performing approaches for sub-8-bit training.
2. The proposed method should be discussed in more detail through a detailed top-level algorithm describing all the operations involved. Please add the pseudocode description of the ShiftQuant algorithm and L1 normalization procedure. Please distinguish clearly between what is novel and what is implemented based on existing methods. It is recommended to add a table that clearly delineates the novel components from existing methods, and suggests specific design decisions you'd like to see explained, such as the rationale behind the power-of-two grouping strategy or the choice of L1 over other normalization approaches.
3. The experimental setup and tool flow used to conduct the experiments should be described in more detail. Please include all the details, such as hardware specifications used for the experiments, software frameworks and versions, training hyperparameters, quantization settings, etc.

1. It is not immediately clear how the ShiftQuant method applies to the matmul, even with the help of Figure 3. Regrouping channels without rearranging the data in memory can only work when the scaling factors are power-of-two as pow2 scaling factors are naturally discrete bins. The paper does not explain directly how the quantization scaling factors are computed. It instead uses "power-of-two grouping", which is misleading. The reader has to infer that the scaling factor is power-of-two.

2. The proposed ShiftQuant method seems not hardware-friendly. Regrouping without memory rearrangement means each element in a quantized dot product has a different scaling factor. To produce a correct output, the hardware matmul unit has to scale each element back before accumulation (step 2 in Figure 3c). In commercial hardware, e.g., GPUs, there is usually no access to the intermediate matmul outputs before accumulation. If building a customized hardware, one scaling factor per channel seems infeasible in terms of chip area and power, even if it is power-of-two.

3. It is not clear how the throughput analysis on GPU is obtained. Line 451 says the performance analysis in Figure 7 is for 6-bit ShiftMM kernel, but NVIDA RTX 3090 does not have 6-bit tensor cores as far as the reader understands.

1. The description of “fully-quantized L1 normalization” lacks clarity. Is the approach intended to replace quantized L2 normalization with a quantized L1 version?
2. While ShiftQuant helps manage a wide gradient range and quantized L1 normalization enables fully quantized normalization layers, these contributions are complementary and show limited direct interaction.
3. The comparison between INT6 and FP16 matrix multiplication on FPGA appears biased. The baseline INT6 format can outperform FP16 in terms of latency, energy, and resource usage, as shown in Table 8. It seems misleading to attribute lower resource usage solely to the proposed method.