NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs

Thank you for your thoughtful review and helpful questions.

1. Doesn’t the SOTA FP8 model work as well as the FP16 model? Why keep both? It is true that state-of-the-art methods such as SmoothQuant have advanced FP8 quantization to the point where it appears nearly lossless on some popular benchmarks. However, recent studies have shown that, when evaluated more thoroughly across a broader range of tasks, even the best-performing 8-bit models still significantly underperform compared to their FP16 counterparts [1]. As an example, human evaluation results from LLM Arena [2] highlight that the 8-bit version of Llama-3.1-405B-Instruct (FP8) exhibits noticeable performance degradation compared to its FP16 counterpart, particularly in tasks involving coding (score: 1277 vs. 1293) and long-context queries (1275 vs. 1282). Note, that Llama-3.1-405B-Instruct (FP8) used SmoothQuant for FP8 quantization [3]. This highlights the need to retain FP16 alongside FP8 to balance inference quality with SLO constraints, as FP8 may still introduce non-negligible errors in some cases.
    
   [1] Tianyi Zhang, Yang Sui, Shaochen Zhong, Vipin Chaudhary, Xia Hu, and Anshumali Shrivastava. 70% size, 100% accuracy: Lossless llm compression for efficient gpu inference via dynamic-length float, 2025.
   [2] https://x.com/lmarena_ai/status/1835760196758728898
   [3] https://arxiv.org/pdf/2407.21783
    
2. PPL Evaluations We report perplexity results on Wikitext2 (W2), C4, and PTB datasets across four models. The FP8 baseline—denoted as (B)—applies per-token activation and per-tensor weight quantization. Our method, NestedFP, is denoted as (N). Additionally, we report results for SmoothQuant, denoted as (S), which builds on (B) with additional activation smoothing.
    
   All quantization schemes were applied to the QKVO and MLP linear layers. For SmoothQuant, we followed the protocol from the original paper and computed activation statistics using the Pile training dataset. A grid search over α ∈ {0.5, 0.7, 0.9} was performed, and the best perplexity value was selected.
    
   Overall, FP8 quantization methods yield slightly higher perplexity than FP16, while SmoothQuant shows comparable performance to the FP8 baseline. The perplexity trends are similar with accuracy evaluations.
    
   | Model | W2(FP16) | W2(B) | W2(N) | W2(S) | C4(FP16) | C4(B) | C4(N) | C4(S) | PTB(FP16) | PTB(B) | PTB(N) | PTB(S) | |------------------|-----------|--------|--------|--------|-----------|--------|--------|--------|-------------|---------|---------|---------| | Llama 3.1 8B | 6.24 | 6.30 | 6.31 | 6.29 | 9.58 | 9.67 | 9.67 | 9.67 | 9.01 | 9.06 | 9.08 | 9.06 | | Mistral Nemo | 5.74 | 5.80 | 5.81 | 5.78 | 9.57 | 9.65 | 9.67 | 9.63 | 8.66 | 8.73 | 8.75 | 8.70 | | Phi-4 | 6.46 | 6.47 | 6.49 | 6.48 | 12.07 | 12.09 | 12.11 | 12.10 | 9.32 | 9.34 | 9.35 | 9.34 | | Mistral Small | 5.29 | 5.35 | 5.35 | 5.34 | 9.09 | 9.20 | 9.21 | 9.20 | 8.00 | 8.07 | 8.07 | 8.06 |  
3. FP16 results for models/tasks in Table 2 FP16 results are provided in Table 1. To improve clarity, we will include them in Table 2 in the future revision.
    
4. Regarding Baseline FP16 Latency of Figure 7 (b) Please refer to Appendix C for the FP16 baseline latency values. In Figure 7(b), “Lvl 3” corresponds to our optimized NestedFP16 kernel. Appendix C also includes a comparison graph for the GEMM shape M×5120×32768, illustrating the performance of both the FP16 baseline and NestedFP16.
    
5. Regarding scaling factor The reviewer is correct that NestedFP uses a fixed scaling factor when encoding the FP16 weight tensor into two FP8 components. This design choice enables exact reconstruction of the original FP16 weights, while maintaining compatibility with FP8 tensor operations.
    
   Importantly, this does not preclude additional scaling: weights may still be scaled as long as the values remain within the representable range of FP8 (E4M3), offering flexibility where needed.
    
   Crucially, our method places no restriction on how activations are quantized. While we adopt per-tensor activation quantization in our experiments, per-token quantization schemes—like those used in the FP8 baseline (B)—can be readily incorporated to further improve accuracy. In this sense, while weight quantization is fixed for exact reconstruction, activation quantization remains fully flexible.
    

We appreciate your thoughtful comments. Below we address your concerns in detail.

1. Regarding PPL evaluations We have additionally ran PPL experiments to show per-channel (per-row) wise weight quantization yields similar perplexity as per-tensor weight quantization. There is no significant difference. W2 denotes Wikitext2 dataset.

 
2. Regarding motivation to switch between FP8 and FP16 models
First and foremost, we would like to clarify that although the 95% confidence intervals of the two models overlap, this does not necessarily imply that the performance difference is negligible. We believe that the degradation observed in the FP8 model (e.g., a 16-point drop in mean performance) can still be considered meaningful, depending on the application context and perspective, and should be taken into account by practitioners when building production-quality systems.
 
That said, we acknowledge that the performance degradation of a well-optimized FP8 model may not be dramatic enough to reach a universal consensus. In this range, subjective judgment can play a role. It is reasonable for some to consider the FP8 model sufficient, and the use of FP16 unnecessary. In response, we would like to emphasize that achieving such a level of FP8 optimization poses practical challenges in real-world production environments.

 
Building a production-grade quantized model is not as simple as achieving good benchmark scores. The model must be robust across a wide range of inputs, without any edge cases where outputs become corrupted. Achieving such robustness often goes beyond applying well-established techniques like SmoothQuant; it typically requires iterative, heuristic tuning, which can be costly and time-consuming.
 
For instance, the Llama-3 technical report, one of the most detailed public documents on FP8 model production, explains that while SmoothQuant serves as the foundation, additional heuristics such as selective quantization and hyperparameter tuning are essential. Without these refinements, the report notes that models may still produce corrupted outputs in certain scenarios, even if their benchmark scores are strong.
 
This refinement process is extremely resource-intensive, despite the heuristics themselves seeming relatively simple. Each tuning step requires thorough testing to identify potential failure cases. The Llama-3 team reports conducting evaluation beyond benchmarks, including case-by-case analysis of over 100,000 responses using a reward model, an auxiliary model trained specifically for quality assessment. Training such a reward model, constructing large-scale evaluation datasets, and analyzing the results all demand substantial engineering effort.
 
Not all organizations have the infrastructure or engineering capacity to support this level of validation. In such cases, relying solely on an FP8 model may be risky and co-deploying an FP16 model alongside the FP8 model can serve as a more robust and pragmatic strategy.

Thank you for raising this thoughtful and important concern.

1. Tradeoff Between Performance Overheads and Memory Savings It is true that NestedFP incurs a small performance overhead (e.g., 3.9% throughput loss) in exchange for memory savings. In response to whether this tradeoff is worthwhile, we emphasize that in LLM inference, memory savings can often be reinvested to improve overall performance. For example, memory savings may enable larger batch sizes—thus increasing throughput—or reduce the frequency of KV cache evictions, which can become a significant bottleneck under memory pressure. While we did not incorporate such reinvestment scenarios in our evaluation in order to isolate the performance impact of kernel slowdown, we believe that, when leveraged, the performance gains enabled by NestedFP’s reduced memory footprint can more than offset the 3.9% throughput loss.
    
   To evaluate how memory savings of NestedFP can be invested to performance improvement, we ran controlled experiments using an H100 GPU with Azure traces (791 requests, 5 minutes) and compared NestedFP16 only deployment (no weight duplication, full KV cache capacity) with a dual deployment of FP16 and FP8 models (weight duplication, KV cache capacity reduced from 34GB → 12GB).
    
   Severe performance degradation was observed when co-deploying both models. P90 TTFT increased from 4.02s → 283.31s, total E2E latency more than doubled (from 300s → 627s), and throughput dropped by > 40%. This was mainly due to the limited batch size and frequent kv cache evictions.  
   In short, the modest kernel slowdown is a worthwhile trade-off, as the resulting memory savings can be reinvested to mitigate key performance bottlenecks, ultimately outweighing the overhead.
    
2. Regarding Presentation Clarity and Table Formatting Thank you for your overall positive comments and thoughtful suggestions. As you pointed out, showing differences in Table 2—similar to Table 1—would improve clarity. We agree and will revise the table to explicitly highlight the differences in FP8 results as well.

We sincerely thank the reviewer for the thoughtful and constructive feedback.  

1. Precision of NestedFP FP16 is a standard data type widely supported by hardware accelerators, including tensor cores. While it is possible to store weights using a 15-bit representation (e.g., by omitting one exponent bit as in FP15), doing so requires custom kernels to reconstruct valid FP16 values at runtime—just as NestedFP does. The key insight is that we're not creating FP15 for its own sake, but rather enabling dual-precision execution from a single representation. To evaluate the general applicability of NestedFP, we provide a detailed analysis of 14 models in Appendix F. These results suggest that a wide range of state-of-the-art models can benefit from NestedFP.
    
2. Memory Issue Justification:Why We Should Save Memory First and foremost, deploying FP8 and FP16 models separately increases memory usage by approximately 1.5× compared to deploying a single FP16 model, which can limit the maximum model size that fits within GPU memory.
    
   Second, in LLM inference, memory capacity is not merely a resource constraint—it has direct implications on performance. Memory savings can often be reinvested to improve system efficiency. For example, memory savings may enable larger batch sizes—thus increasing throughput—or reduce the frequency of KV cache evictions, which can become a significant bottleneck under memory pressure. While we did not incorporate such reinvestment scenarios in our evaluation in order to isolate the performance impact of kernel slowdown, we believe that, when leveraged, the performance gains enabled by NestedFP’s reduced memory footprint can more than offset the throughput loss caused by kernel slowdown (i.e., 3.9%).
    
   To evaluate how memory savings of NestedFP can be invested to performance improvement, we ran controlled experiments using an H100 GPU with Azure traces (791 requests, 5 minutes) and compared NestedFP16 only deployment (no weight duplication, full KV cache capacity) with a dual deployment of FP16 and FP8 models (weight duplication, KV cache capacity reduced from 34GB → 12GB). Severe performance degradation was observed when co-deploying both models. P90 TTFT increased from 4.02s → 283.31s, total E2E latency more than doubled (from 300s → 627s), and throughput dropped by > 40%. This was mainly due to limited batch sizes and frequent KV cache evictions.
    
   In short, the modest kernel slowdown is a worthwhile trade-off, as the resulting memory savings can be reinvested to mitigate key performance bottlenecks, ultimately outweighing the overhead.
    
3. Evaluation on Large Models/Reasoning Models We evaluated the effectiveness of NestedFP on two 70B-scale models: LLaMA 3.1 70B, a general-purpose language model, and DeepSeek-R1-Distill-LLaMA-70B, which is specialized for reasoning tasks.
    

• Applicability: We analyzed the weight distributions of LLaMA 3.1 70B and DeepSeek-R1-Distill-70B to assess the applicability of NestedFP across QKVO and MLP linear layers. For both LLaMA 3.1 70B and DeepSeek-R1-Distill-70B, over 93% of layers (523/560 and 521/560, respectively) satisfied the constraint. While the remaining layers are left in FP16, most layers can be converted to the NestedFP format. This demonstrates that NestedFP is applicable to both large-scale models and reasoning models.
• Accuracy: We benchmarked both models on three tasks: Minerva MATH, BBH, and MMLU Pro. We compared FP16 results, FP8 baseline (per-token activation and per-tensor weight quantization), and our NestedFP8 format. Results on the table show that NestedFP8 consistently matches or outperforms the FP8 baseline across benchmarks, while maintaining accuracy very close to full-precision FP16. These results confirm that NestedFP8 is effective for both large-scale and reasoning models.

• Latency: We measured inference latency on LLaMA 70B using H100 GPUs with tensor parallelism (TP=2). NestedFP8 achieves average x1.43 TTFT speedup over FP16 across different input lengths, while NestedFP16 incurs minimal overhead (<4% increase). These results confirm that the NestedFP format overhead remains negligible at large scale, while FP8 quantization provides significant speedup as GEMM operations dominate the computation.

4. Detailed Accuracy Comparison To perform a per-layer analysis of the decoder, we quantized all QKVO and MLP FP16 linear layers into FP8 and measured the relative error across decoder layers using the Pile training dataset. Results for selected layers (L10, L20, L30, L40) and the full-model average show that NestedFP8 generally yields slightly higher error than DeepSeek FP8 due to its coarser quantization. As NestedFP8 applies only to linear layers, attention-layer quantization schemes like DeepSeek FP8 can still be applied independently without conflict.

5. Scheduling Replicate First and foremost, we would like to emphasize that the primary goal of our work is to enable per-iteration precision switching (every 10–100 ms) between FP8 and FP16 without model duplication. NestedFP is scheduler-agnostic and can be seamlessly integrated with any LLM scheduling policy. While the design of such schedulers is beyond the scope of this paper, we consider developing effective scheduling strategies for NestedFP a promising direction for future research.
    
   Nevertheless, to provide a rough estimate of the potential performance benefits in a realistic serving scenario, we implemented a simple load-aware scheduler on the vLLM v1 engine. This scheduler selects NestedFP8 when the batch token count exceeds half of max_num_batched_tokens, and defaults to NestedFP16 otherwise—achieving adaptive precision switching without forecasting or user input. Note that this is a very naive policy, and many optimization opportunities remain on the table, such as incorporating per-query SLO constraints, KV cache utilization, or more sophisticated load prediction mechanisms.
    
   We conducted experiments on a single-node H100 80GB server using Mistral Small 24B as the test model Azure LLM trace. In addition to TTFT and TPOT, we evaluated SLO attainment by defining two SLO classes following the latency analysis framework suggested by SOLA (MLSys'25). Specifically, we set tight SLO as ≤ 5× the average single-request latency and loose SLO as ≤ 10× the average single-request latency, where the single-request latency was measured in our experiment environments.
    
   Even with simple policy, being able to dynamically switch between FP8 and FP16 led to dramatic latency improvements over static FP16-only serving: p90 TTFT decreased from 2.42s → 0.89s (x0.36), p90 TPOT decreased from 0.26s → 0.063s (x0.24), tight SLO attainment increased from 49.9% → 95.1% (x1.90) and loose SLO attainment increased from 89.5% → 100% (x1.12).
    
   These results demonstrate the effectiveness of token-level precision switching, highlighting the motivation behind NestedFP, which is to enable it in a memory-efficient manner.

First, we would like to clarify that NestedFP does not incur any additional memory overhead compared to a standard FP16-only deployment. While the simple scheduling policy used in our proof-of-concept experiment bases decisions solely on system load, it is true that some scheduling policies may take memory size into account for various reasons. However, even under such policies, NestedFP does not lead to any scheduling inefficiencies related to memory usage.

In response to your request for additional results, we conducted the same experiment described in Answer #5, this time using a 4×H100 distributed setup with the Llama 70B model. The results under the highest tested load (7.47 req/s) show that our approach significantly outperforms the FP16-only baseline: p90 TTFT decreased from 12.06s → 1.66s (x0.14). Tight SLO attainment increased from 0% → 50.3%. Loose SLO attainment increased from 5.9% → 85.9% (x14.5). The full results across all tested request rates are summarized below:

These results confirm that our approach—choosing between NestedFP8 and NestedFP16 based on system load—yields consistent, significant gains even for a large model in a distributed setting. We believe this addresses your final concern and will incorporate the findings into our manuscript. Thank you.

We sincerely thank the reviewer for the detailed and insightful feedback.

1. Evaluation on Large Models/Reasoning Models We evaluated the effectiveness of NestedFP on two 70B-scale models: LLaMA 3.1 70B, a general-purpose language model, and DeepSeek-R1-Distill-LLaMA-70B, which is specialized for reasoning tasks.
    

• Applicability: We analyzed the weight distributions of LLaMA 3.1 70B and DeepSeek-R1-Distill-70B to assess the applicability of NestedFP across QKVO and MLP linear layers. For both LLaMA 3.1 70B and DeepSeek-R1-Distill-70B, over 93% of layers (523/560 and 521/560, respectively) satisfied the constraint. While the remaining layers are left in FP16, most layers can be converted to the NestedFP format. This demonstrates that NestedFP is applicable to both large-scale models and reasoning models.
• Accuracy: We benchmarked both models on three tasks: Minerva MATH, BBH, and MMLU Pro. We compared FP16 results, FP8 baseline (per-token activation and per-tensor weight quantization), and our NestedFP8 format. Results on the table show that NestedFP8 consistently matches or outperforms the FP8 baseline across benchmarks, while maintaining accuracy very close to full-precision FP16. These results confirm that NestedFP8 is effective for both large-scale and reasoning models.

• Latency: We measured inference latency on LLaMA 70B using H100 GPUs with tensor parallelism (TP=2). NestedFP8 achieves average x1.43 TTFT speedup over FP16 across different input lengths, while NestedFP16 incurs minimal overhead (<4% increase). These results confirm that the NestedFP format overhead remains negligible at large scale, while FP8 quantization provides significant speedup as GEMM operations dominate the computation.

2. Regarding the question on reading FP8 tensors in inference In our implementation, the two 8-bit tensors (upper and lower) are stored as separate, fully contiguous FP8 checkpoints, each aligned in memory to prevent scattered access. During FP8 inference, only the upper tensor is loaded, and since it is accessed contiguously, it incurs no additional L1/L2 cache misses compared to native FP8 checkpoints.
    
3. How to support BF16 weights? If a model is released in BF16, it can be cast to FP16 during preprocessing and subsequently converted into NestedFPweights. The error introduced by this casting is usually negligible in practice. To demonstrate it, we conducted an element-wise comparison between the FP16-cast versions of LLaMA 3.1 8B and Qwen3 14B and their original BF16 counterparts and found out that not a single parameter in any layer showed a numerically meaningful deviation (≥ 1e-10 absolute or relative error). This negligible deviation is also reflected in the perplexity evaluations shown in the table below. Across all cases, casting to FP16 has minimal impact, supporting the use of FP16 as a preprocessing step for NestedFP.

4. Throughout reduction of NestedFP8 compared to Torch FP8 We used a custom CUTLASS kernel for FP8 evaluation. The observed performance gap is likely due to suboptimal kernel configuration (e.g., tile shapes), as we did not perform an extensive kernel tuning process—FP8 kernel optimization was not the primary focus of this work. However, based on your comment, we revisited this issue and found that there is no reason to use a custom kernel for NestedFP8; the same kernel used in the baseline can be applied. We will update our evaluation accordingly in future revisions and expect no performance gap for FP8 GEMM.
    
5. Latency (TTFT and TPOT)
   We implemented a simple load-aware scheduler on the vLLM v1 engine. This scheduler selects NestedFP8 when the batch token count exceeds half of max_num_batched_tokens, and defaults to NestedFP16 otherwise—achieving adaptive precision switching without forecasting or user input. We conducted experiments on a single-node H100 80GB server using Mistral Small 24B as the test model Azure LLM trace.

6. Regarding DFloat11 DFloat11 leverages unused exponent bits in bfloat16 and introduces an efficient inference kernel. While both DFloat11 and NestedFP exploit unused exponent bits and employ custom kernels, their goals and effects differ. DFloat11 focuses on compressing model weights and using the saved memory to achieve performance gains, particularly in comparison to CPU offloading scenarios, without any loss in accuracy. In contrast, NestedFP does not reduce the actual model size. Instead, it supports both lossless FP16 inference and quantized FP8 inference to better meet SLO constraints. As mentioned earlier, even with a simple scheduler, NestedFP can deliver substantial latency improvements over static FP16-only serving.
    
7. Regarding possible extensions to FP4 We appreciate the suggestions regarding both FP4 extensions. Since (1) FP4 requires more careful handling of scaling, and (2) FP8 offers very few unused exponent bits to exploit, extending similar ideas to FP4 is non-trivial—but still a direction worth exploring.

Thank you for the positive feedback and for maintaining your recommendation. We’ll incorporate the new experiments and discussions into the final version of the manuscript.