LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

W2: Enhancements in memory-constrained deployment scenarios.

Yes, LoRC demonstrates significant benefits in memory-constrained deployment scenarios. To further validate this, we provide additional experiments here.

• Deployment on Consumer GPUs (24GB Memory).

Table R1: LLaMA-3-8B-Instruct model on a single GPU with 24GB memory.

Table R2: LLaMA-3-70B-Instruct model on 8 GPUs with 24GB memory.

These results demonstrate that LoRC enables the deployment of models that would otherwise be impossible to run with full KV cache on consumer GPUs.

• Performance Preservation Under Aggressive Compression.

In this set of experiments, we compare LoRC with token-eviction method (H2O) [1] and quantization method (KIVI) [3], using LLaMA-3-8B-Instruct. For H2O and LoRC, we keep the same 60% KV cache budget. For KIVI, we use the KIVI-4-bit implementation. We evaluate accuracy on BoolQ and OpenBook QA, and Rouge-Lsum for XSum.

These experiments demonstrate that LoRC not only enables deployment in memory-constrained environments but also maintains higher model performance compared to alternative approaches.

• Additionally, as discussed in our first response to reviewer CNeC, LoRC improves system throughput while achieving these memory savings.

[3] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.

We appreciate the reviewers’ insightful feedback.

W1.1: Details on implementation for computing the attention layer.

Response: Within the attention layer, there are two major differences compared to the standard implementation. First, for the caching stage, we project keys to low-rank space without position embedding and store only the compressed representation. Second, for the attention computation, we load compressed keys and reconstruct them, apply RoPE, and then compute attention scores with position-aware keys.

To deal with the compatibility with RoPE, the most straightforward implementation -- applying RoPE for each compressed key -- brings computational overhead. To address this, we follow H2O [1] and FastGen [2] to develop a customized kernel that fuses cache reconstruction and rotation operations. This approach minimizes memory transfers and computational overhead, enhancing the overall throughput.

Specifically, we leverage Triton to implement a fused kernel that optimizes memory access and computation. With this kernel, we are able to combine key reconstruction from compressed representation and apply RoPE rotations on-the-fly. It also streamlines memory access because the single data load from global memory for compressed keys requires minimal data movement.

With these engineering efforts, we achieve higher throughput compared to baseline configurations.

W1.2: Speedup compared to the full KV cache method.

Response:

Yes, we are able to achieve speedup compared to the full KV cache method. We present the throughput analysis below.

Following H2O [1], we conducted throughput experiments with fixed input and output sequence lengths using the LLaMA-3-70B-Instruct model on a node equipped with eight NVIDIA H100 80GB HBM3 GPUs. As discussed above, our method involves additional computations to recover the compressed cache and manage RoPE, and we developed a customized kernel that fuses cache reconstruction and rotation operations.

The results above indicate that our engineered efforts can streamline the attention computation with LoRC compression, thereby achieving higher throughput compared to the full cache scenario.

We also compare LoRC with other methods using the metric seconds per iteration (s/it) on XSum to obtain the real-world throughput. We adhered to the same 60% KV cache budget for both H2O and LoRC. In an effort to demonstrate compatibility with other KV cache compression methods, we integrated LoRC with both H2O and standard 8-bit quantization. Specifically, we allocated a 70% KV cache budget to LoRC and 85% to H2O, resulting in approximately a 60% overall cache budget. A similar configuration was used for the combination of LoRC with 8-bit quantization. For LoRC with quantization, we also allocate 70% KV cache budget to LoRC and 85% to 8-bit quantization.

Our results show that LoRC achieves superior performance compared to existing compression methods while maintaining competitive throughput. While the SVD-based computations introduce some overhead, LoRC still has a higher throughput than its full cache baseline because we deployed a customized kernel to streamline the attention computation. It's important to note that our primary objective was not to develop a Pareto-optimal method (simultaneously optimizing both performance and throughput), but rather to introduce an orthogonal approach to KV cache compression and establish the progressive compression strategy.

Additionally, the integration of LoRC with both token-eviction and quantization methods demonstrates its compatibility with existing compression methods. With only a slight performance trade-off (11.0/10.9 vs 11.2), these hybrid configurations achieve better throughput compared to using LoRC alone, which shows the potential of combining LoRC with orthogonal compression approaches.

[1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023.
[2] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs, ICLR 2024.

Q3: Progressive Strategy and Method Effectiveness.

3. Single-layer and Inter-layer Analysis:
   The inter-layer dependency means we consider the layer sensitivity from a network-wide perspective rather than analyze each layer alone, and we carefully consider the inter-layer dependencies throughout this paper. Theoretically, Section 5.3 provides a mathematical proof of error propagation patterns to illustrate how the inter-layer dependencies guide our design of the progressive compression strategy. Empirically, Section 6.6 especially validates the error propagation effects -- the noise caused by shallow compressed layers will be amplified more after propagation through the network.

4. Superiority of the Progressive Compression Strategy:
   Similar to the first point made in this question, previous methods need calibration data for truncation rank searching (ASVD) or post-compression finetuning (FWSVD), while our compression strategy works directly without any calibration data or post-compression model tuning. Additionally, the empirical results demonstrate that LoRC can preserve competitive performance compared to the full cache baseline with a significant compression ratio.

Q4.1: Experiments on models beyond llama.

We conduct more experiments using the OPT-30B and OPT-66B models on XSum and OpenBook QA (OBQA) to provide a more comprehensive evaluation of our method. We report Rouge-Lsum on XSum and accuracy on OBQA. The results are as shown below, where the percentage denotes the KV cache budget.

The results demonstrate that:

• Performance Preservation with Significant KV Cache Saving: Compared to the full-cache baseline, LoRC maintains model performance with negligible degradation even at significant KV cache compression levels.
• Broad Applicability: Beyond the main results reported in our paper, these additional experiments demonstrate that LoRC is broadly applicable across diverse models, including LLaMA-2-13B, LLaMA-3-8B-Instruct, LLaMA-3-70B-Instruct, OPT-30B, and OPT-66B.

Q4.2: Comparison with Other KV Cache Compression Methods

Response: We compare LoRC with the token-eviction method (H2O) and the quantization method (KIVI) using LLaMA-3-8B-Instruct. For H2O and LoRC, we keep the same 60% KV cache budget. For KIVI, we use the KIVI-4-bit implementation. We evaluate accuracy on BoolQ and OpenBook QA, and Rouge-Lsum for XSum. The results below show LoRC can preserve a better performance compared to token-eviction and quantization methods when KV cache is aggressively compressed.

For throughput comparison, we compare LoRC with other methods using the metric seconds per iteration (s/it) on XSum. We adhered to the same 60% KV cache budget for both H2O and LoRC. To demonstrate compatibility with other methods, we integrated LoRC with both H2O and standard 8-bit quantization. Specifically, we allocated a 70% KV cache budget to LoRC and 85% to H2O, resulting in approximately a 60% overall cache budget. A similar configuration was used for the combination of LoRC with 8-bit quantization. For LoRC with quantization, we also allocate 70% KV cache budget to LoRC and 85% to 8-bit quantization.

Our results show that LoRC outperforms other methods while maintaining competitive throughput. While the SVD-based computations introduce some overhead, LoRC still has a higher throughput than its full cache baseline because we deployed a customized kernel to streamline the attention computation. Note that our primary objective was not to develop a Pareto-optimal method (simultaneously optimizing both performance and throughput), but rather to introduce an orthogonal approach to KV cache compression and establish the progressive compression strategy.

Additionally, the integration of LoRC with both token-eviction and quantization methods demonstrates its compatibility with existing compression methods. With only a slight performance trade-off, these hybrid configurations achieve better throughput compared to using LoRC alone, which shows the potential of combining LoRC with orthogonal compression approaches.

We appreciate your feedback and thoughtful review of our work.

Q1: Previous SVD works claim performance loss without updating the weight matrix, why the performance loss in this work is minimal?

Response: Our work differs fundamentally from previous SVD works [1, 2, 3] in two key aspects:

• Scope of compression: [1, 2, 3] compress model parameters including more general weight matrices in LLMs, our KV cache compression approach specifically targets the Key and Value weight matrices in the attention module. Compressing only Kv matrices preserves the model's computational patterns in other components.
• Consideration of network-wide effects: previous works analyze each layer in isolation, while we consider the critical inter-layer dependencies. This explains why they report performance loss when the cumulative effects are ignored when compressing the matrices.

Our key insight is that compression errors propagate differently through the network depending on where they originate. Our theoretical analysis (Section 5.3) rigorously proves that errors from earlier layers get amplified through network propagation, even if these layers individually show low sensitivity. That is why previous SVD approaches, which only consider single-layer properties, require weight updates or retraining to maintain performance. In contrast, our method's minimal performance loss stems from carefully accounting for these propagation effects through progressive compression and cumulative sensitivity analysis.

[1] ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
[2] Language model compression with weighted low-rank factorization
[3] SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

Q2: Why cumulative condition number is reasonable for rank selection?

Response: The details of the derivation and illustration of cumulative condition number can be found in lines 188- 221. To summarize here, the cumulative condition number serves as a mathematically principled criterion for rank selection by capturing layer sensitivity from a network-wide perspective. Our approach is grounded in the following derivation:

• Local Sensitivity: As shown in Eq. (11), a layer's acceptable input perturbation is inversely proportional to its condition number $\kappa(A_l)$. This provides a natural measure of a single layer sensitivity.

• Error Propagation: In multi-layer transformers, compression-induced perturbations propagate through subsequent layers. By computing the product of condition numbers from the current layer through all subsequent layers (Eq. (12)), the cumulative condition number provides a holistic measure of each layer's impact on the final output.

This formulation of the cumulative condition number is theoretically justified by our analysis in Section 5.3. It proves that errors in earlier layers pass through more subsequent transformations and nonlinearities, thus being amplified more significantly. The cumulative condition number directly quantifies this effect, making it an ideal metric for guiding compression decisions.

Q3: Progressive Strategy and Method Effectiveness.

This question consists of multiple points, let us address each concern:

1. Different Focus and Theory Compared to Prior Works:
   While previous works (e.g., ASVD, FWSVD) compress entire model parameters, we specifically target KV weight matrices for cache compression. Unlike prior approaches that analyze layers in isolation, we consider the inter-layer dependencies from the network view. Our strategy is built on the intuition that compressed shallow layers could lead to cascading errors that propagate and amplify through the network, which is also supported by our theoretical analysis as presented in Section 5.3.
   Note that ASVD requires calibration data for perplexity calculation to determine layer sensitivity, and FWSVD requires post-compression finetuning. Our method only needs to access the model weights to determine layer sensitivity and work directly without post-compression model tuning -- this makes our approach more practical and widely applicable.

2. Whether Compress Final Layer Aggressively:
   The final layer’s importance could come from multiple components within that layer. Note that we only compress the key/value weight matrix of each layer. In Figure 4, we show that these matrices can be approximated with low reconstruction error, including the key/value weight matrix of the final layer. It means that compressing the final layer's key/value weight matrix via low-rank approximation is practical. Moreover, according to our derivation of the progressive compression strategy (Eq. 16), the error of this layer has the least propagation in the forward pass, thus it can be compressed most aggressively.

W3 & Q1: How do you handle RoPE?

Response:
RoPE does bring additional computations to LoRC. To address this, we follow H2O [3] and FastGen [5] to develop a customized kernel that fuses cache reconstruction and rotation operations. This approach minimizes memory transfers and computational overhead, enhancing the overall throughput.

Specifically, we leverage Triton to implement a fused kernel that optimizes memory access and computation. With this kernel, we are able to combine key reconstruction from compressed representation and apply RoPE rotations on-the-fly. It also streamlines memory access because the single data load from global memory for compressed keys requires minimal data movement.

In a complete RoPE workflow, there are two stages

• Caching: We project keys to low-rank space without position embedding and store only the compressed representation.
• Attention computation: Within the customized kernel, we load compressed keys into shared memory, reconstruct and apply RoPE in a fused operation, and then compute attention scores with position-aware keys.

With these engineering efforts, we achieve higher throughput compared to baseline configurations.

[5] Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs, ICLR 2024.

W2 & Q2: Latency & Throughputs Experiments.

Response:
Before addressing the latency and throughput results, we would like to emphasize that compressing KV cache itself is meaningful, because it enables the deployment of models that would otherwise be impossible to run with full KV cache on resource-constrained scenarios (e.g., consumer GPUs). In the main results of this paper, we aim to demonstrate that LoRC can preserve competitive performance compared to the full cache baseline when compressing aggressively.

We present latency analysis and throughput experiments below.

Latency Analysis:

The latency analysis is shown in the above table. Note that sensitivity calculation only takes once for a given model, and the SVD processing is also a one-time implementation during the model initialization stage. Such latency will not affect inference speed. Compared to the whole inference duration, the latency incurred by these processes is negligible. For example, the LLaMA-3-70B-Instruct model requires approximately 1 hour and 8 minutes to process 1,000 summaries from the XSUM dataset with a batch size of 32 and a sequence length of 8,000. The combined latency introduced by sensitivity calculations and SVD processing represents only 0.6% of the total inference time.

Throughput Analysis:

Following H2O [3], we conducted throughput experiments with fixed input and output sequence lengths using the LLaMA-3-70B-Instruct model on a node equipped with eight NVIDIA H100 80GB HBM3 GPUs. Because our method involves additional computations to recover the compressed cache and manage RoPE, we developed a customized CUDA kernel that fuses cache reconstruction and rotation operations. This approach minimizes memory transfers and computational overhead, enhancing the overall throughput.
The results below indicate that our engineering efforts can streamline the attention computation with LoRC compression, thereby achieving higher throughput compared to full cache scenarios.

We also compare LoRC with other methods [3, 4] using the metric seconds per iteration (s/it) on XSum to obtain the real-world throughput. For throughput analysis, we compared LoRC with other methods using the metric seconds per iteration (s/it) on XSum. We adhered to the same 60% KV cache budget for both H2O and LoRC. To demonstrate compatibility with other KV cache compression methods, we integrated LoRC with both H2O and standard 8-bit quantization. Specifically, we allocated a 70% KV cache budget to LoRC and 85% to H2O, resulting in approximately a 60% overall cache budget. A similar configuration was used for the combination of LoRC with 8-bit quantization. For LoRC with quantization, we also allocate 70% KV cache budget to LoRC and 85% to 8-bit quantization.

Our results show that LoRC achieves superior performance compared to existing compression methods while maintaining competitive throughput. While the SVD-based computations introduce some overhead, LoRC still has a higher throughput than its full cache baseline because we deployed a customized kernel to streamline the attention computation. It's important to note that our primary objective was not to develop a Pareto-optimal method (simultaneously optimizing both performance and throughput), but rather to introduce an orthogonal approach to KV cache compression and establish the progressive compression strategy.

Additionally, the integration of LoRC with both token-eviction and quantization methods demonstrates its compatibility with existing compression methods. With only a slight performance trade-off (11.0/10.9 vs 11.2), these hybrid configurations achieve better throughput compared to using LoRC alone, which shows the potential of combining LoRC with orthogonal compression approaches.

[3] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023.
[4] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.

W1: Comparison with Related Work.

Response:
While we appreciate the reviewer highlighting these relevant works, we respectfully disagree with this criticism regarding comparisons with [1] and [2]. According to ICLR 2025 guidelines (https://iclr.cc/Conferences/2025/ReviewerGuide), authors are "not required to compare their own work to" very recent papers or non peer-reviewed (e.g., ArXiv) papers that are: a) Published after July 1, 2024 ;b) Not peer-reviewed; c) Only available on arXiv. Both [1] and [2] fall in the above categories. Actually, both of them are also under review of ICLR 2025. It would have been impossible for us to compare with these concurrent submissions.

Nevertheless, we are open to a discussion of these approaches. Our method differs from [1] and [2] in two fundamental aspects:

• Data-free Compression: ASVD [1] requires calibration data for perplexity calculation; PALU [2] needs a dataset to compute Fisher information. Our method requires only the model's weight matrices to determine layer sensitivity -- this makes our approach more practical and widely applicable.

• Analysis of Inter-layer Dependencies: Both [1] and [2] analyze layer sensitivity in isolation. Our key innovation is considering inter-layer dependencies from a network-wide perspective. This is crucial because compression-induced perturbations in multi-layer transformers propagate through subsequent layers. Our progressive compression strategy provides a holistic measure of each layer's impact on the final output, and we theoretically prove its effectiveness through rigorous error propagation analysis (Section 5.3).

These distinctions highlight our unique contributions.

[1] ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models, arxiv 2024.
[2] Palu: Compressing KV-Cache with Low-Rank Projection, arxiv 2024.

Thank you so much for your engagement in this discussion period. We organize our responses below.

C1: Comparison with concurrent work.

We thank reviewer 9w6H for withdrawing their concern about concurrent work comparisons. Rather than evading discussion, we welcome the opportunity to include these relevant concurrent works, as they help highlight two key advantages of our approach:

• First, LoRC achieves data-free compression without any calibration data or profiling data for determining compression extents [1, 2]. Instead, LoRC operates purely on weight matrices, which makes it immediately deployable, especially in scenarios where additional data access is restricted or computationally expensive.
• Second, different from the related works that analyze layer sensitivity in isolation, we consider inter-layer dependencies from a network-wide perspective and propose the progressive compression strategy accordingly. We also provide a theoretical analysis of error propagation through transformer layers that rigorously support this progressive design, as acknowledged by reviewer vHmh, P7NJ, and CNeC in their mentioned strengths.

C2: GPU Implementation, Throughput Performance, and Writing.

This comment consists of several points, we organize our response in the following aspects.

Clarification Request: Before addressing specific points, we would like to seek clarification on what is meant by "SVN" - we assume this may refer to SVD, which is the technique used in our work.

Paper Positioning and Core Contributions: We want to emphasize that we are not positioning our paper as a throughput-SOTA compression method, as SVD-based compression inherently introduces computational overhead. Instead, our key contribution lies in the design of an orthogonal compression technique, the progressive compression strategy, the corresponding theoretical foundations, and its ability to preserve performance under aggressive compression.

Throughput Performance: As shown in our throughput analysis made in the first-round response (https://openreview.net/forum?id=NI8AUSAc4i&noteId=21SkfMVh8A), while our Triton-based implementation helps LoRC achieve 1.14-1.26× speedup over the full cache baseline, it still faces inherent computational overhead compared to throughput-optimized methods. However, our experiments reveal LoRC's strong compatibility with both token-eviction and quantization methods. To pursue throughput enhancement, the integration with H2O or 8-bit quantization achieves better throughput (195.9 s/it and 193.7 s/it vs 203.4 s/it) with minimal performance impact (11.0/10.9 vs 11.2). Regarding the comment "It is tricky to get this kernel to perform well with high performance," we believe future work can either explore compression methods integration or more optimized kernel-level implementation to further advance the throughput.

Terminology Clarification and Writring Focus: Regarding terminology, we acknowledge the confusion in using CUDA kernel and Triton kernel interchangeably. To clarify, our implementation uses Triton for kernel development. It is great to know such implementation draws interest from this reviewer, but our original manuscript focused on LoRC's core algorithmic and theoretical contributions. Following the reviewer's suggestion, we are revising the paper to include these engineering efforts and to demonstrate LoRC's compatibility with existing throughput-optimized methods.

We again sincerely appreciate the reviewer's engagement in the discussion stage. Please let us know if there are any additional concerns or questions.

Q2: Comparison with Other KV Cache Compression Methods

Response: We aimed to include all three suggested works in our comparison. However, we successfully incorporated only H2O due to limitations with the other methods. ZipCache, which has not yet released the full code necessary for running inference, was replaced by KIVI [2], another method that employs KV cache quantization. SVD-LLM, which includes a calibration stage using additional data to update the compressed matrix, was deemed non-comparable to our experimental setting due to this difference.

We conduct experiments using LLaMA-3-8B-Instruct. For H2O and LoRC, we keep the same 60% KV cache budget. For KIVI, we use the KIVI-4-bit implementation. We evaluate accuracy on BoolQ and OpenBook QA, and Rouge-Lsum for XSum. The results below show LoRC can preserve a better performance compared to token-eviction and quantization methods when KV cache is aggressively compressed.

For throughput analysis, we compared LoRC with other methods using the metric seconds per iteration (s/it) on XSum. We adhered to the same 60% KV cache budget for both H2O and LoRC. In an effort to demonstrate compatibility with other KV cache compression methods, we integrated LoRC with both H2O and standard 8-bit quantization. Specifically, we allocated a 70% KV cache budget to LoRC and 85% to H2O, resulting in approximately a 60% overall cache budget. A similar configuration was used for the combination of LoRC with 8-bit quantization. For LoRC with quantization, we also allocate 70% KV cache budget to LoRC and 85% to 8-bit quantization.

Our results show that LoRC achieves superior performance compared to existing compression methods while maintaining competitive throughput. While the SVD-based computations introduce some overhead, LoRC still has a higher throughput than its full cache baseline because we deployed a customized kernel to streamline the attention computation. It's important to note that our primary objective was not to develop a Pareto-optimal method (simultaneously optimizing both performance and throughput), but rather to introduce an orthogonal approach to KV cache compression and establish the progressive compression strategy.

Additionally, the integration of LoRC with both token-eviction and quantization methods demonstrates its compatibility with existing compression methods. With only a slight performance trade-off (11.0/10.9 vs 11.2), these hybrid configurations achieve better throughput compared to using LoRC alone, which shows the potential of combining LoRC with orthogonal compression approaches.

[2] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.

Q3: Different trends in sensitivity by compression between training and inference.

We appreciate the reviewer's insightful observation about the contrasting sensitivity trends between training and inference. This is because forward propagation amplifies early-layer compression errors during inference, while backward propagation amplifies later-layer compression effects during training.

To provide a detailed discussion, we can compare the error propagation mechanisms. In our method (inference-stage compression), errors from compressed earlier layers propagate forward through the network. As shown in our Theorem 3 (Section 5.3), the error at network output is bounded by Eq.16, where earlier layer errors get amplified more because they pass through more subsequent transformations and nonlinearities.

For those training-stage compression methods as introduced in [4-6], their rationale is that gradient errors propagate backward through the network. For example, DropBP [6] measure the layer sensitivity by: $S_l = \sum_i ( \|\nabla \mathbf{W}_i \|_2 - \|\nabla \mathbf{W}_i^{(l)} \|_2 )^2$ where $S_l$ denotes the sensitivity of the l-th layer. Here, $\nabla \mathbf{W}_i$ represents the parameter gradient of the $i$-th layer when no layers are dropped, while $\nabla \mathbf{W}_i^{(l)}$ denotes the parameter gradient of the $i$-th layer when the $l$-th layer is dropped during backward propagation. It shows that later layers have higher impact on gradients propagating backward to all previous layers.

We will add this discussion to the appendix and add the suggested works in related works.

We appreciate the reviewer’s insightful comments, we organize our responses as below.

Q1: Latency and Throughput Experiments

Response:

Latency Analysis:

The latency analysis is shown in the above table. Note that sensitivity calculation only takes once for a given model, and the SVD processing is also a one-time implementation during the model initialization stage. Such latency will not affect inference speed. Compared to the whole inference duration, the latency incurred by these processes is negligible. For example, the LLaMA-3-70B-Instruct model requires approximately 1 hour and 8 minutes to process 1,000 summaries from the XSUM dataset with a batch size of 32 and a sequence length of 8,000. The combined latency introduced by sensitivity calculations and SVD processing represents only 0.6% of the total inference time.

Throughput Analysis:
Our method involves additional computations to recover the compressed cache and manage RoPE. To address this, we developed a customized kernel that fuses cache reconstruction and rotation operations. This approach minimizes memory transfers and computational overhead, enhancing the overall throughput.

Following H2O [1], we conducted throughput experiments with fixed input and output sequence lengths using the LLaMA-3-70B-Instruct model on a node equipped with eight NVIDIA H100 80GB HBM3 GPUs. The results below indicate that our engineering efforts can streamline the attention computation with LoRC compression, thereby achieving higher throughput compared to full cache scenarios.

[1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023.