70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)

We sincerely thank the reviewer for the thoughtful feedback and for recognizing the insightfulness of our findings and the effectiveness of our approach. Below, we address the concerns raised in your review.

[W1] Evaluation on LLM is not very compreshensive.

We present additional evaluations of BF16 and DF11 models on various benchmarks in the table below. Since DF11 models produce bit-for-bit identical outputs to their BF16 counterparts, their performance on any benchmark is exactly the same. Figure 13 further illustrates the identical outputs generated by DF11 and BF16 models.

[W2] Latency overhead

We agree that the non-zero decompression overhead may impact latency-sensitive applications at small batch sizes. However, as shown in Figure 6, this overhead becomes negligible in large batch size settings due to amortization. Additionally, we plan to explore further hardware-aware optimizations (such as fusing the decompression and GEMM kernels) in future work to reduce latency overhead.

We thank the reviewer for giving insightful feedback and for highlighting the clarity and contribution of our work. We address the comments and questions below.

[W1] Performance benchmark for a single layer

Below, we provide additional performance benchmarks for a single transformer layer from Llama 3.1 8B Instruct, evaluated at various token batch sizes. Additional results for single layer decompression can be found in Figure 7, which presents the decompression throughput and latency for various matrices ranging from 4096x2048 to 4096x65536 in size.

[W2] Quantization of bfloat16 models to more than 8 bits is a niche problem

Our work addresses a different goal from quantization methods. While quantization introduces a trade-off between accuracy and compression, our approach guarantees bit-identical outputs to the original bfloat16 model. This can be valuable in scenarios where exact numerical accuracy is critical or where the model's performance is highly sensitive to quantization.

[W3,W4,W5] Figure font size and the capitalization of bfloat16

We appreciate the reviewer's careful review of our paper. We will revise the figures in the camera-ready version to improve readability. We will also add a note to acknowledge the alternate capitalization of bfloat16.

[W6] Source code

We will release the source code upon paper acceptance.

[Q1] Can DFloat11 be enabled on-demand?

Yes, enabling DFloat11 on demand is feasible. Since our method compresses only the exponent bits while leaving the sign and mantissa unchanged, the compressed exponents can be offloaded to the CPU and transferred to the GPU to save memory when needed.

[Q2] Fusing DF11 decompression with GEMM

Fusing DF11 decompression into GEMM or attention kernels is a promising direction that could further reduce decompression overhead. This approach would involve grouping compressed exponent values into tiles or blocks, which each thread block can load and decompress in SRAM. The decompressed weights could then be used directly on chip for matrix multiplications, avoiding the bottleneck of an additional write and read of decompressed weights to and from HBM. We thank the reviewer for this insightful suggestion and plan to explore it in future work.

We thank the reviewer for providing valuable feedback and for highlighting the novelty of our work and the effectiveness of our method. We address your concerns and questions below.

[W1] Performance comparison between DF11 and AWQ/GPTQ/FP8

We present additional accuracy comparisons between BF16, DF11, FP8, and GPTQ (INT4). As shown in the table below, FP8 and GPTQ introduce non-zero accuracy degradation on various benchmarks, while DF11 maintains identical accuracy to BF16. Moreover, even in cases where the accuracy remains roughly unchanged, the quantized model's behavior can diverge significantly from that of the original model, as discussed in Line 44 of the paper.

We also provide throughput results (prefill 2K tokens, decode 1K tokens) for DF11, BF16, and GPTQ (INT4) with different batch sizes below. These results show that while DF11 incurs additional decompression overhead, it remains practical across batch sizes. Importantly, DF11 guarantees accuracy and behavior that are exactly identical to the original model, whereas quantized models (such as FP8, GPTQ, AWQ, etc.) may experience accuracy loss or behavioral changes.

[W2, Q2] Time consumption of compression

We report detailed compression times in Appendix F. Compressing an 8B model takes approximately 5 minutes, while a 405B model requires about 4 hours. Compression is performed only once per model and is fast and efficient. Since each transformer block can be compressed independently, we parallelize the process across multiple CPU cores to further accelerate compression.

[W3,Q1] Applying DFloat to FP8/FP4 models

Dynamic-length float can be applied to FP8 and FP4 weights to enable lossless compression for these formats. Similar to BF16, we apply Huffman coding to compress the exponent bits while leaving the sign and mantissa uncompressed. Since FP4 and FP8 have fewer exponent bits than BF16, the resulting lookup tables are smaller, which may allow for faster decompression. The compression gains vary across these formats, as the effectiveness depends on the entropy of the weights. For example, we measured the entropy of FP8 weights in Qwen3-8B to be 6.78 bits, corresponding to a 15.3% compression ratio. In contrast, FP4 weights in Llama-3.3-70B have an entropy of 3.97 bits (a 0.8% compression ratio), likely too small to be impactful. We thank the reviewer for this suggestion and plan to explore it further in future work.

[W4] Relevance of Large Token Batch Sizes and Decompression Overhead

While we agree that typical serving batch sizes for decoder-only LLMs are usually limited to 16-32, larger token batch sizes are relevant in other important scenarios. For example, when using LLMs as embedding models (where no KV cache is involved) or in diffusion models for image and video generation, the models process a large number of tokens at once. In these settings, the decompression overhead is effectively amortized and becomes negligible. As shown in Table 3, DF11 inference latency for diffusion models increases by only 4-5% compared to BF16.

For smaller batch sizes (e.g., 16–32), the additional overhead is not a fixed 70%. Instead, it is the constant overhead required to decompress the model from DF11 to BF16. Figure 10 provides a detailed comparison of inference efficiency between BF16 and DF11 across various batch sizes and models.

[Q3] Latency for different batch sizes

Please refer to Figures 4 and 10 for latency measurements across batch sizes of 1, 2, 4, 8, 16, and 32. As the batch size doubles, throughput approximately doubles while latency increases slightly.

We thank the reviewer for providing thoughtful feedback, and for recognizing the motivation of our work and the effectiveness of our proposal. We address your concerns and questions below.

[W1, Q1] Novelty and Contributions

While Huffman coding optimizations for GPUs have been studied by previous works, lossless model compression that supports GPU inference is relatively unexplored. Most previous work has focused on lossy compression (such as quantization) or lossless methods intended only for reducing storage on disk, rather than for use during inference. Supporting GPU inference of losslessly compressed models requires non-trivial algorithmic designs and engineering efforts. Our key contributions include:

1. Practical solution for deployment: We bridge the gap between the low-entropy observation and real-world deployment by designing a practical method for online inference using losslessly compressed weights. This includes proposing transformer-block-level decompression to achieve better throughput on GPUs.
2. Proven benefits: Our results demonstrate that significant compression gains can be achieved with minimal loss in GPU inference speed, while preserving losslessness.

We believe our work opens up new directions for lossless model compression in online inference and can inspire further progress in efficient model deployment.

[W2] Comparison with INT8 Accuracy and Throughput

We have added additional accuracy comparisons between BF16, DF11, and INT8 models. The INT8 models use GPTQ-W8A8 quantization with the llm-compressor library. Our results show that BF16 and DF11 achieve identical accuracy across all tasks. In contrast, the INT8 models show a non-zero drop in accuracy compared to BF16/DF11, which can be as high as 6.37% on GSM8K with quantized Gemma-3-27b-it. We will include further throughput comparisons between BF16, DF11, and INT8 models in the final version of the paper. According to the SmoothQuant paper, INT8 quantization can achieve up to 1.35x to 1.42x speedup over 16-bit inference.

[Q2] Extension to Other Formats (FP16, FP8, and 8-bit Quantized Models)

Our method is also applicable to FP16, FP8, and other 8-bit quantized formats. The table below shows the entropy of Qwen3-8B weights in FP16, FP8, and INT8 formats, along with their corresponding compression factors. Dynamic-length float encoding can be extended to support these formats to enable their lossless compression.