HiFC: High-efficiency Flash-based KV Cache Swapping for Scaling LLM Inference

We appreciate the reviewer’s thorough assessment of our work and the constructive suggestions provided. Our detailed responses to the raised points are given below.

Q4.1. How does HiFC perform and scale in realistic multi-GPU and multi-SSD deployments?

A4.1. We appreciate the reviewer’s insightful comment regarding the scalability of HiFC. As stated in our paper’s limitations, we have extended HiFC to support multi-GPU and multi-SSD configurations and confirmed its scalability with new experiments.

HiFC supports the following topologies:

• 1:1 (GPU:SSD) – baseline configuration for workstation-level applications
• N:1 – multiple GPUs sharing one SSD, minimizing SSD cost
• N:N – multiple GPUs with multiple SSDs (RAID-enabled), maximizing both GPU performance and SSD bandwidth

Experimental Setup

Extended Experimental Validation

Key Findings:

• HiFC with a 1:1 configuration already outperforms DRAM-based swapping by 5% TPS while providing much larger KV cache capacity.
• 2:1 scaling achieves ~2× throughput, as swap overlaps rarely collide on a single SSD.
• 2:2 scaling maintains the 2× throughput increase while expanding request capacity.
• For multi-GPU setups, the pSLC space for KV cache is evenly assigned to each GPU, consistent with the cache management strategy used in vLLM.
• Adding more SSDs increases request capacity. Select SSDs according to LLM size.

Conclusion:

These results confirm that HiFC scales effectively across multi-GPU and multi-SSD setups, preserving both throughput and memory capacity advantages.

Q4.2. Does HiFC maintain its performance across different model sizes and dynamic, mixed-length workloads?

A4.2. We appreciate the reviewer’s question regarding HiFC’s performance under mixed workloads and varying model sizes. HiFC was designed with real-world deployment in mind and has been evaluated on diverse datasets during development. The paper used a fixed model and input/output length to ensure a fair comparison with DRAM-based swapping. For additional verification, we conducted new tests using multiple popular open-source models (DeepSeek-Llama-8B, DeepSeek-Qwen-14B, and Mistral-7B) and datasets with context lengths ranging from a few hundred to over 18K tokens. These tests were run with the same GPU/SSD configuration as in the paper.

Experimental Setup

Extended Experimental Validation

Key Findings

• Across all tested models and datasets, the TPS difference between DRAM and HiFC remained within ~2%.
• Results are consistent with those reported for DS-Qwen-32B in the paper.
• Mistral’s 32K limit affected some NarrativeQA requests, but performance trends remained similar.

Conclusion:

These additional experiments confirm that HiFC maintains DRAM-comparable throughput across diverse models and dynamic context lengths. Therefore, HiFC is robust enough to replace DRAM in long-context LLM inference services under realistic mixed workloads.

Q4.3. How does HiFC compare to InstInfer in terms of cost, latency, and compatibility?

A4.3. We thank the reviewer for suggesting a comparison with InstInfer, an excellent work that leverages Computational Storage Devices (CSDs) to optimize KV cache management and reduce I/O bottlenecks. Both InstInfer and HiFC aim to extend KV cache to NAND Flash, but they adopt fundamentally different approaches:

• InstInfer is a hardware-specialized solution using CSDs to process KV cache with in-storage attention, eliminating PCIe bottlenecks.
• HiFC is a software-general solution built on the widely used vLLM framework, utilizing commodity GPUs and SSDs to achieve cost efficiency and compatibility.

The differences in cost, latency/throughput, and compatibility are summarized below.

Cost Comparison

InstInfer requires CSD hardware (e.g., Daisyplus OpenSSD or SmartSSD), which significantly increases the initial cost. In contrast, HiFC uses readily available pSLC SSDs, resulting in a much lower TCO and easier deployment.

Latency / Throughput Comparison

HiFC hides swap latency via pipeline scheduling and uses GDS to minimize CPU/DRAM involvement. InstInfer achieves higher gains when fully utilizing CSD’s internal bandwidth but is more hardware-dependent.

Compatibility

InstInfer demonstrates strong scalability but relies on specialized hardware and custom software. HiFC, as a software-level solution, is broadly compatible with existing LLM frameworks and hardware, making it easier to deploy at scale.

Conclusion

The additional experiments show that HiFC achieves throughput on par with DRAM across diverse LLM architectures and varying context lengths, while reducing the total cost of ownership by 4.5× through the use of an inexpensive pSLC SSD instead of costly DRAM. These findings highlight HiFC’s practicality and cost-efficiency, demonstrating its suitability for long-context LLM inference in real-world deployments.

We have implemented these features and will release the full code and results on GitHub before the camera-ready version.

We appreciate the reviewers’ time and consideration in reviewing our rebuttal.

We appreciate the reviewer’s thorough assessment of our work and the constructive suggestions provided. Our detailed responses to the raised points are given below.

Q3.1. Will the swapping-based approach used in HiFC introduce unacceptable latency overheads in latency-oriented serving systems?

A3.1. We truly appreciate the reviewer’s insightful question regarding whether HiFC’s swapping approach could introduce problematic latency in latency-sensitive serving systems. This is an important aspect, and we carefully examined it during our evaluation.

We assumed GPU VRAM alone couldn’t handle the immense request load.

As with any swapping mechanism, moving KV cache from GPU HBM to another memory tier introduces some overhead. In vLLM, this is handled through two strategies:

• Swapping, which adds I/O overhead during KV cache transfer.
• Recompute, which avoids I/O but incurs additional computation.

To keep latency under control, the vLLM scheduler selectively sacrifices a few requests to free up HBM, preventing overall service delays. HiFC adopts the same scheduling logic, and our results show that it maintains end-to-end throughput and request latency comparable to DRAM-based swapping.

Experimental Setup

• GPU: A100 (80GiB)
• HBM KV cache size: 4 GiB (starvation condition)
• DRAM KV cache size: 50 GiB
• SSD for HiFC: 1TB (pSLC 200 GiB)
• Model: DeepSeek-R1-Distill-Llama-8B
• Dataset: THUDM/LongBench/gov_report_e (avg. 8K length)

Evaluation Results

What We Observed

• When HBM is limited, both swapping and recompute mechanisms are triggered as expected.
• HiFC successfully hides SSD swap latency through pipeline overlap, minimizing the impact on service performance.
• Interestingly, HiFC delivered 5% higher throughput than DRAM-based swapping, likely because DRAM contention can increase tail latency in some cases.
• Overall, HiFC achieved latency very close to DRAM while supporting 4× more requests.

Conclusion

Our evaluation suggests that HiFC’s design effectively prevents SSD latency from becoming a bottleneck. By leveraging vLLM’s scheduling and overlapping techniques, HiFC hides most of the swap cost and preserves low latency. These findings give us confidence that HiFC can be applied to latency-sensitive inference scenarios without introducing major concerns.

Q3.2. Are the price ranges and other components used in the TCO analysis, particularly those in Table 1, sufficiently justified with citations given their critical role in supporting the paper’s cost reduction claims?

A3.2. We apologize for not including explicit citations for the price ranges and components in our original TCO analysis. As cost and energy claims are central to the paper, we re-evaluated all parameters using updated market data. The original paper used prices from the first half of 2025, while the table below reflects data from the second half of 2025 with added references for clarity.

Updated 3-years TCO Table

Reference

• Memory, SSD: Memory.NET / Amazon
• Power cost, PUE: 2024 US Data Center Energy Usage Report

Key Findings

• The updated analysis shows that the 3-year TCO gap has widened: DRAM is now 6.1× more expensive than HiFC (previously 4.5×).
• This increase is mainly due to recent DRAM price hikes, further strengthening the cost advantage of SSD-based HiFC.
• All cited prices are derived from publicly available sources, ensuring reproducibility of the analysis.

Conclusion

We regret not including these references in the original submission. The updated data confirms that our TCO claims remain valid and that the cost benefits of HiFC have become even more pronounced in the current market environment. This reinforces the paper’s main argument that HiFC provides a practical and economically significant solution for large-scale LLM inference deployments.

Q3.3. What are the potential systems engineering challenges in maintaining SSD-based systems, particularly regarding increased failure rates?

A3.3. We thank the reviewer for raising concerns about maintaining SSD-based systems. SSDs in HiFC are managed like standard GPU server drives with NVMe logs for monitoring, and failures have minimal impact as drives are easily replaced. Using pSLC regions, HiFC also ensures long SSD lifespan.

NVMe Error Management and Monitoring Technologies

• SMART Log: Shows health data (temperature, errors, remaining life) for early failure detection.
• Error Log: Records I/O and PCIe errors to identify and troubleshoot faults.
• Self-Test Log: Runs diagnostics to detect hidden issues before they impact performance.
• Telemetry Log: Collects detailed metrics (I/O, wear) for tuning and reliability checks.

Conclusion

HiFC adds no extra risk beyond standard SSD-managed GPU servers. Using NVMe monitoring and quick drive replacement, system stability is maintained, and SSD-related engineering challenges remain minimal.

Q3.4. Are the price trends for these technologies (DRAM versus SSD) such that the TCO benefits from this method will shrink, persist, or grow over the coming years?

A3.4. We apologize for not including explicit citations for the price ranges and components in our original TCO analysis. We appreciate the reviewer’s observation, as cost and energy claims are central to our work and require clear justification.

For this rebuttal, we have re-examined current market prices (H2 2025) and projected future trends for GPUs, DRAM, and SSDs. The updated values confirm that the cost gap between DRAM and SSD has widened, further strengthening HiFC’s economic advantage.

Projected 3-Year Price and TCO Trends (2025–2027)

Thanks to the reviewer’s question, we re-examined DRAM/SSD price trends and found SSDs’ TCO advantage has increased due to rising DRAM costs. While GPUs like Blackwell require expensive DDR5, HiFC uses SSDs only for temporary KV cache, avoiding high-cost memory. Future high-performance SSDs will further enhance offloading, and HiFC is already compatible via GDS.

Conclusion:

The updated market data confirms that HiFC’s TCO benefits are not only valid but are likely to become even more significant over the coming years, reinforcing its practical value for large-scale LLM deployments.

Q3.5. Are there any sustainability concerns from repeatedly burning through flash drives (even if this has lower TCO)?

A3.5. We appreciate the reviewer’s question on SSD sustainability, as endurance affects both cost and environmental factors. HiFC ensures long-term reliability by using only the pSLC region, which has over 10× the endurance of TLC NAND. In our 1 TB SSD, the 200 GiB pSLC region supports 6,000 TB writes—8× the rated 750 TBW. HiFC also uses sequential I/O to reduce internal GC operations, further extending lifespan.

Experimental Setup

• GPU: A100 (80GB)
• SSD for HiFC: 1TB (pSLC 200 GiB)
• Datasets: GovReport (avg. 8.7K)
• Output length: 1K

Predicted SSD Lifetime Based on Testing and Lifetime Calculation Formula

• Total KV written per day (TiB) = KV size per test (GiB) × Tests per day
• Lifetime (years) = pSLC TBW (TiB) / (Total KV Written per Day (TiB) × 365)
• Lifetime (years) = 6000 / (1.98 × 365) ≈ 8.3

Key Findings

• Under our test conditions, HiFC’s SSD would last up to 8.3 years, far exceeding typical service lifetimes.
• For comparison, reports such as Tom’s Hardware estimate the expected service life of GPUs used for LLM inference at around 3 years.
• This means the SSD would only need to be replaced once while the GPU may undergo three replacements in the same period.

We have implemented these features and will release the full code and results on GitHub before the camera-ready version.

We appreciate the reviewers’ time and consideration in reviewing our rebuttal.

We appreciate the reviewer’s thorough assessment of our work and the constructive suggestions provided. Our detailed responses to the raised points are given below.

Q2.1. Is the paper missing a fair comparison with other DRAM-based offloading methods (FlexGen, DeepSpeed, etc.)?

A2.1. We thank the reviewer for raising the concern about the lack of explicit comparison with other DRAM-based offloading methods. Existing offload-based frameworks, including FlexGen[6] and DeepSpeed[9], rely on host DRAM to store KV cache, but their behavior differs from that of vLLM[5].

Offloading vs. Swapping: Key Differences

• FlexGen and DeepSpeed store KV cache in host DRAM continuously and fetch it directly during attention.
• vLLM keeps KV cache in GPU HBM and only swaps out to host DRAM when necessary, hiding most latency through its scheduler.

We would like to emphasize that a direct, one-to-one comparison between the offloading and swapping techniques is inherently difficult, since they rely on different inference frameworks. Even when using the same GPU only, the performance of vLLM and DeepSpeed can differ substantially, as each framework employs fundamentally different methods for attention processing and scheduling. Therefore, we compared the DRAM‐based swapping technique implemented in vLLM with our HiFC method, which is also implemented in vLLM. By keeping the framework constant, we isolated the effect of storage type (DRAM vs. SSD), eliminating confounding factors and clearly demonstrating the benefits of HiFC, which builds on vLLM while replacing DRAM with SSDs to achieve comparable latency at a significantly lower cost.

Q2.2. Does HiFC enable maintaining larger KV cache and context than DRAM-based offloading?

A2.2. We thank the reviewer for this important question. Yes, HiFC enables maintaining a significantly larger KV cache and supports longer contexts compared to DRAM-based offloading. HiFC builds on vLLM’s swapping mechanism but replaces host DRAM with SSDs, which greatly expands storage capacity while keeping latency low. Under the same TCO as DRAM, HiFC can store over 6.2× more KV cache.

Memory Scaling for KV cache

Table below compares request capacity and relative TCO across different memory configurations.

• Requests = Memory Size / avg. KV cache size

• For the cost of DRAM, operators can provision four SSDs, enabling up to 6.2× more requests compared to DRAM
• The pSLC region of a 1 TiB SSD (200 GiB) can be configured as a 800 GiB pSLC with multiple SSDs in RAID0.
• With vLLM’s flexible allocation, HiFC fully exploits this expanded Flash cache without requiring model modifications.

In practice, configuring four SSDs in HW/SW RAID0 and registering the Flash cache file in HiFC enables seamless scaling, allowing service providers to handle longer contexts and more requests while maintaining significantly lower costs than DRAM-based offloading. Conclusion

These results demonstrate that HiFC not only matches DRAM-based approaches in performance but also provides a much larger KV cache capacity at lower cost, directly addressing the reviewer’s concerns on scalability and impact.

Q2.3. Is the 4.5× TCO saving from HiFC significant when GPU ownership costs dominate the overall expenses in cloud environments?

A2.3. We agree with the reviewer that in cloud environments, GPU rental costs overshadow DRAM or SSD ownership costs. Under such conditions, our 4.5× TCO savings for KV cache management may appear less significant compared to the high monthly cost of GPUs.

However, HiFC targets a different deployment model: LLM service providers operating on-premise GPU infrastructure. In these environments, operators must optimize memory expansion costs rather than GPU rental fees when scaling to handle larger KV caches and longer contexts.

• For cloud users, scaling typically means renting more GPU instances, where costs are dictated by the provider’s pricing model.
• For on-premise providers, scaling can be achieved by adding low-cost SSDs to existing servers, avoiding expensive GPU upgrades.
• SSDs are simple to deploy, can reuse existing hardware, and enable quick scaling without downtime.

In this context, HiFC provides a valuable alternative: rather than purchasing an additional $14,000 A100 GPU**, an operator can **expand KV cache capacity** with a **$100 SSD, achieving substantial scalability at minimal cost.

Cost Comparison (3-Year Ownership vs. Cloud Rental)

Key Takeaways

• For cloud services, GPU rental dominates total costs, making memory-related savings less impactful.
• For infrastructure owners, HiFC provides a highly cost-efficient scaling method, replacing costly GPU upgrades with inexpensive SSDs.
• HiFC allows seamless KV cache expansion within vLLM without architectural changes, making it a practical and economically meaningful solution beyond the context of cloud rentals.

Conclusion

While HiFC’s TCO savings may be negligible in cloud rental scenarios, it becomes highly significant for service providers managing their own GPU infrastructure. The ability to expand memory using commodity SSDs instead of costly GPUs provides a compelling advantage in real-world deployments.

Q2.4. Does the DRAM-free HiFC design deliver meaningful benefits and practical impact beyond the added SSD component, given that GPU ownership costs dominate the overall TCO?

A2.4. We thank the reviewer for this important question. Yes, HiFC provides meaningful benefits and practical impact that go far beyond simply adding an SSD, even when GPU ownership costs dominate overall TCO. HiFC was specifically designed to address the high cost and limited capacity of DRAM in KV cache management, offering a scalable and cost-effective solution without compromising performance.

Summary of HiFC’s Contributions

Additional Remarks

Beyond cost savings, HiFC introduces a new design principle for KV cache scaling, utilizing commodity SSDs to overcome memory limits without architectural modifications. It extends the widely adopted vLLM framework, making it immediately usable by both researchers and practitioners. The full implementation will be released as open source to encourage adoption.

In real-world deployments, instead of upgrading to an additional $14,000 A100 GPU (80 GiB), **operators can simply add a$100 SSD (1 TiB), expanding KV cache capacity by up to 12.8×** with minimal modifications to their systems.

Conclusion

These results show that HiFC is not just a DRAM replacement using SSDs, but a practical, scalable, and cost-efficient solution to the KV cache scalability problem. We believe HiFC makes a meaningful contribution to enabling economical, high-performance long-context LLM inference, directly addressing the reviewer’s concerns regarding its impact and applicability.

We have implemented these features and will release the full code and results on GitHub before the camera-ready version.

We appreciate the reviewers’ time and consideration in reviewing our rebuttal.

We appreciate the reviewer’s thorough assessment of our work and the constructive suggestions provided. Our detailed responses to the raised points are given below.

Q1.1. How will the system scale in a multi-GPU environment?

A1.1. We thank the reviewer for raising the important question about HiFC’s scalability in multi-GPU environments. To address this concern, we have extended HiFC to operate efficiently on multi-GPU and multi-SSD configurations and conducted additional evaluations to verify its behavior under these setups.

HiFC supports three deployment topologies:

• 1:1 (GPU:SSD) – baseline configuration commonly used in workstation setups
• N:1 – multiple GPUs sharing a single SSD to minimize storage cost
• N:N – multiple GPUs connected to multiple SSDs (with RAID) to maximize both performance and bandwidth

Experimental Configuration

Extended Evaluation Results

Observations

• The 1:1 configuration already provides 5% higher TPS than DRAM while expanding KV cache capacity.
• In the 2:1 configuration, throughput nearly doubles (~2×) as the number of GPUs increases because swap I/O collisions on the SSD are rare.
• In the 2:2 configuration, throughput also achieves a near-2× increase while additionally supporting a higher number of requests.
• In multi-GPU configurations, each GPU is allocated the same pSLC region for KV cache, following vLLM’s cache management policy.
• The maximum request capacity scales with the number of SSDs. Choose SSD count based on service scale.

Conclusion

These experiments demonstrate that HiFC scales robustly in multi-GPU and multi-SSD environments, achieving near-linear throughput scaling without sacrificing memory capacity.

Q1.2. Can the authors provide results on other LLMs to demonstrate broader applicability?

A1.2. We thank the reviewer for asking about HiFC’s applicability to different LLMs and workloads. HiFC was developed with practical deployment scenarios in mind and has been tested with diverse datasets during development. In the original paper, we intentionally used a fixed model and input/output lengths to provide a fair and controlled comparison against DRAM-based swapping. To further demonstrate generality, we conducted additional evaluations using several widely adopted open-source models (DeepSeek-Llama-8B, DeepSeek-Qwen-14B, and Mistral-7B) across datasets with context lengths ranging from a few hundred tokens to over 18K tokens. All experiments were performed under the same GPU/SSD configuration as in the paper.

Experimental Setup

Extended Evaluation Results

Key Observations

• Across all tested models and datasets, the performance gap between DRAM and HiFC remains within ~2%, confirming negligible overhead.
• These findings align with the results presented for DS-Qwen-32B in the paper.
• The 32K context limit of Mistral affected some NarrativeQA requests, but the overall trend remained consistent.

Conclusion

The additional experiments demonstrate that HiFC consistently delivers DRAM-comparable throughput across multiple LLM architectures and dynamic context lengths, while achieving a 4.5× reduction in total cost of ownership by replacing expensive DRAM with a low-cost pSLC SSD. This confirms that HiFC is broadly applicable and effective for long-context LLM inference in realistic deployment scenarios.

We have implemented these features and will release the full code and results on GitHub before the camera-ready version.

We appreciate the reviewers’ time and consideration in reviewing our rebuttal.