(W1): In the Palu framework, a full GEMM is required to rematerialize each key and value matrix to be used in the attention module. This adds a lot of complexity, particularly when decoding a large output sequence length.

Ans: We acknowledge that computational overhead is a critical challenge when replacing full-dimensional KV-Cache with low-rank latent representations. However, we respectfully disagree with the characterization of this as a weakness. Addressing this challenge is central to Palu’s design, and we have proposed techniques such as G-LRD and custom GPU kernels to optimize the tradeoff between computation and memory efficiency. These innovations enable Palu to significantly reduce memory usage while accelerating LLM decoding. For a detailed explanation, we encourage the reviewer to refer to our global response.

(W2). What the paper is really doing is not KV cache compression, but really compressing the QKV WxA layer preceding attention.

Ans: We respectfully disagree with the reviewer’s statement, as it mischaracterizes the focus of our work. While Palu achieves KV-Cache hidden-dimension reduction by decomposing the weight matrices of the Key and Value layers, its design and objectives are fundamentally different from weight compression. If the goal were weight compression, we could store KV-Cache in the original hidden dimension, eliminating the need for runtime reconstruction. However, Palu specifically targets KV-Cache compression, which necessitates caching latent tensors and dynamically reconstructing them during inference - a process far more complex than static weight compression.

To address these challenges, Palu incorporates advanced techniques such as G-LRD, rank search, quantization integration, and custom GPU kernels to balance memory reduction and computational overhead effectively. These design choices are intrinsic to KV-Cache compression and not applicable to weight compression alone. We encourage the reviewer to refer to our global response for a detailed explanation of Palu’s unique focus and design.

(W3) The experimental setup doesn't make a lot of sense. It is mentioned that FlashAttention is not used for apples-to-apples comparisons. But FlashAttention anyway is there to speed up the prefill phase where the KV cache doesn't exist yet. Why does it then matter? Why are you including prefill latency when your solution is supposed to be addressing KV cache compression, which is used in the decoding phase?

Ans: The latency results we reported in our paper are the decoding latency, not the prefill latency, as the reviewer pointed out that KV-Cache is used in the decoding stage. We have updated our manuscript to further clarify this part, which you can view in the revised submission.

(Q1) In Table 1, why does Llama2-7B have better perplexity and accuracy than Llama3-8B-Inst? There is definitely something wrong with the experimental setup.

Ans: Thank you for raising this concern. However, the observation that Llama-2-7B achieves better perplexity than Llama-3-8B on WikiText-2 is consistent with existing literature[1] and not indicative of an issue in our experimental setup. One potential explanation is that a larger vocabulary (128k for Llama-3-8B versus 32k for Llama-2-7B) inherently increases prediction difficulty for each token, resulting in a naturally higher perplexity. It is also important to note that Llama-3-8B-Inst demonstrates improved zero-shot accuracy over Llama-2-7B, with average scores of 67.99 and 65.05, respectively. To ensure reproducibility, we have provided code implementation in revision. We hope this explanation addresses your concern.

[1] SpinQuant: LLM quantization with learned rotations: https://arxiv.org/pdf/2405.16406

(Q2): Rather than compressing weights, since KV cache compression is desired, can a method such as ESPACE [1] be used to project activations to a low rank directly?

Ans: Thank you for the suggestion. Since ESPACE was published on October 7, 2024, the same date as our submission deadline, we could not include a discussion or comparison in our initial submission.  Broadly, ESPACE can serve as an alternative approach for deriving low-rank projection matrices. Palu derives low-rank matrices $\mathbf{W}\approx\mathbf{A}\mathbf{B}$ through weight decomposition. ESPACE performs calibration on input x and then uses eigenvector decomposition to obtain the low-rank projection matrix $\mathbf{P}$. During inference, ESPACE operates as $\mathbf{h}= \mathbf{x}\mathbf{A'}$ and $\mathbf{y}=\mathbf{h}\mathbf{B'}$ with $\mathbf{A'}=\mathbf{P}$ and $\mathbf{B'}=\mathbf{W}\mathbf{P}$ pre-computed, eventually achieving the same down-projection and reconstruction style of forwarding pass as the weight decomposition style adopted in Palu.

It would be interesting in future work to compare the accuracy differences by deriving a low-rank projection matrix via decomposing weight directly or other techniques. Furthermore, we note that our proposed GPU kernel optimizations and quantization-friendly enhancements—such as fusing the Hadamard Transform—are agnostic to the algorithm to derive a low-rank projection matrix. Thus, these techniques can provide a solid foundation for further research integrating ESPACE or other methods.

[1] ESPACE: Dimensionality Reduction of Activations for Model Compression https://arxiv.org/abs/2410.05437

Thank you for your follow-up. We believe our previous responses, including the global response, have comprehensively addressed your concerns. For Weakness 1, we strongly encourage you to refer to the expanded discussion in the global response, which explains how Palu leverages the compute-memory trade-off to achieve speedups despite the additional computational requirements.

For reviewer convenience, we include Table R3, extracted from Figure 6 of our paper, with extended coverage to test sequence lengths up to 256k. The results demonstrate that Palu consistently achieves significant speedups (1.51x ~ 1.95x) across various sequence lengths.

Table R3. Latency evaluation of computing attention score. All time measurements are reported in microseconds.

For Weakness 2, which pertains to the conceptual understanding of our paper, we note that a global response is also provided to clarify Palu’s distinction from related works focusing on parameter reduction. Regarding the remaining concerns related to the experimental setup and results (e.g., perplexity results), clarifications and references have also been provided.

If there are specific areas or additional “concrete evidence” you feel are missing, we kindly request clarification so we can further support our rebuttal and address your concerns effectively.

We are glad to hear that our responses to the questions were well-received. Yes, we have included a discussion of ESPACE in Appendix K of the revised manuscript, as suggested. Additionally, thank you for providing further clarification regarding your concern about the additional overhead of GEMM required by Palu. Please refer to the subsequent comments for a detailed discussion.

The Explanation Behind Palu's Speedup

We thank the reviewer for your insightful comments. As the reviewer noted, GEMM involves additional FLOPs compared to GEMV, as it processes a stack of vector multiplications. The reviewer mentioned that the GEMM operation is needed in our proposed Palu framework. However, we emphasize that Palu significantly reduces the global memory (DRAM) read/write traffic, which typically dominates the latency of attention operations in LLM decoding and, therefore, can still achieve high speedups.

Why Palu Requires Fewer Global Memory Access. In the baseline approach, the attention scores for each head are computed directly via $\mathbf{q}_i\mathbf{K}_i^T$, which is a GEMV with $\mathbf{q}_i\in\mathbb{R}^{1\times d_h}$ and $\mathbf{K}_i \in \mathbb{R}^{L\times d_h}$ denote the query and full-rank key matrix, respectfully. In Palu, we store the low-rank latent $\mathbf{H_i}\in \mathbb{R}^{L\times r}$ and the reconstruction matrix $\mathbf{B_i}\in \mathbb{R}^{r\times d_h}$ in place of the full rank key. The full-rank Key matrix $\mathbf{K}_i$ can then be reconstructed via $\mathbf{H}_i\mathbf{B}_i$.

Instead of naively reconstructing $\mathbf{K_i}$ with a GEMM operation followed by a GEMV to compute attention scores, the Palu framework employs tiling and operator fusion to streamline the process into a single optimized operation. For a detailed description of our kernel designs and optimizations, please refer to Appendix B. This approach avoids materializing the full $\mathbf{K_i}$ in global memory by keeping intermediate activations (e.g., reconstructed $\mathbf{K_i}$) in the GPU’s shared memory and registers. Consequently, Palu significantly reduces global memory access, requiring only $\mathbf{q_i}$, $\mathbf{H_i}$, and $\mathbf{B}$ - which are substantially smaller compared to $\mathbf{q_i}$ and $\mathbf{K_i}$ in the baseline method.

Quantitive Analysis. To provide a detailed comparison, we analyze the global memory traffic (reads/writes) and FLOPs required for attention score computation for the baseline GEMV approach and Palu, considering the different decomposition granularities proposed in our paper: M-LRD, G-LRD, and J-LRD. The quantitative results are summarized in Table R5. For this evaluation, we set a 75% compression rate for the key matrix, as the setup used in our paper, and a sequence length of 32k. The latency values average over 100 runs, with global memory traffic measurements taken using the NVIDIA Nsight Compute profiler.

Table R5. Global memory (off-chip) footprint, FLOPs, and latency evaluation on calculating attention score.

From Table R5, it is evident that Palu significantly reduces global memory traffic, achieving up to a 75% reduction compared to the baseline GEMV approach. This reduction directly translates to latency improvements, as memory access is the primary bottleneck in attention computation. For example, in the M-LRD case, memory traffic decreases by nearly 4x, leading to proportional latency improvements of 3.78x. This demonstrates that even with additional computation, significant speedups can be achieved by optimizing memory access.

However, as the reviewer noted, additional FLOPs could harm the latency. For the J-LRD case, where reconstruction FLOPs are highest, the latency increases by nearly 4.77x compared to the baseline. This reflects the trade-off between reconstruction cost and latency improvements. The G-LRD variant balances accuracy and efficiency by grouping heads, achieving a 1.6x speedup while maintaining moderate computational overhead.

In conclusion, while Palu introduces additional computation from reconstruction, its significant memory traffic reduction, combined with the G-LRD design and optimized reconstruction kernel with tiling and operator fusion, makes it an effective solution for memory-bound LLM decoding. We appreciate the reviewer’s feedback, which helped us clarify these advantages. For further validation, we have uploaded a detailed report from the Nsight Compute System, as suggested.

(W1): In the Palu framework, a full GEMM is required to rematerialize each key and value matrix to be used in the attention module. This adds a lot of complexity, particularly when decoding a large output sequence length.

We would like to further clarify a misunderstanding from this comment. In Palu, there is no need to rematerialize the full Value matrix, whether for RoPE-based or non-RoPE attention. This is because the reconstruction matrix for the Value can be fused with the next linear layer's weight (see Sec 3.1.1, Lines 185–201), enabling a highly efficient inference process for the Value. Therefore, the reconstruction process described in the previous comment is required only for the RoPE-based Key Cache.

We hope that this further clarification, together with the demonstrated practical speedups discussed previously, could further address your concern regarding the additional GEMM requirements for performing reconstruction in the proposed Palu framework.

Thank you again for your continued engagement and for raising your score. We sincerely appreciate the time and effort you have dedicated to reviewing our work.

We believe that the experimental results presented in Table R6, along with the discussion on the compute-bound and memory-bound nature of the operations, directly address the previously raised concerns on the additional GEMM-type operation. Also, we proved that our approach can still practically bring speedups.

If there are specific aspects of our results or arguments that you find unconvincing, we would greatly appreciate your clarification. Gaining a deeper understanding of your concerns will allow us to address them more effectively and refine our presentation for future paper readers. We also welcome any further questions or clarifications that may help address other parts of the work where you feel additional discussion is needed.

Thank you for your response and for raising this important distinction. We want to clarify that we do NOT assert all computations are inherently memory-bound. Factually, Table R5 from our previous response demonstrates that LLM decoding with low-rank compressed KV-Cache reconstruction can become compute-bound, potentially leading to slowdowns. Consequently, this observation motivated us to propose a novel G-LRD design to address this issue effectively.

To further illustrate, we have expanded Table R5 into Table R6 with estimated data movement and compute times, calculated using the peak global memory bandwidth and compute throughput of the RTX 4090. We classify an operation as memory-bound if its latency is dominated by memory access time, and compute-bound if it is determined by computational throughput [1, 2]. These estimates highlight that GEMV operations in the baseline and M-LRD remain memory-bound. However, with Palu’s G-LRD and J-LRD, the workload shifts to being compute-bound. Importantly, by leveraging modern GPUs’ high compute throughput and maintaining a balanced computational overhead, G-LRD achieves a 1.6x speedup over the baseline.

We hope this additional clarification addresses your concern, and we are happy to provide further explanations if needed.

Table R6: Comparison of total memory traffic (W+R), FLOPs, estimated compute and memory times, and measured average latency for baseline and Palu's different decomposition methods.

[1] LLM Inference Unveiled: Survey and Roofline Model Insights https://arxiv.org/abs/2402.16363

[2] Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures https://dl.acm.org/doi/10.1145/1498765.1498785

We sincerely thank the reviewers for their time and effort throughout the discussion session.

As the discussion cycle concludes today, we want to summarize the experiments and corresponding discussions regarding the introduction of additional GEMM operations in Palu:

1. Comprehensive latency measurement of attention score calculation with reconstruction on various sequence lengths. On benchmarking attention score computed with additional GEMM required for reconstruction, our evaluations comprehensively evaluate the sequence length from 4k to 256k (Table R3). Also, we set the tensor configurations (e.g., hidden dimensions, number of attention heads) identical to the parameters of Llama-2 to ensure benchmarking is aligned with real-world LLM usage.

2. Quantitative Analysis. The detailed analysis in Table R6 provides insights into memory footprint reduction, validated using the Nsight Compute profiler, and additional FLOPs quantified for each decomposition scenario (e.g., M-LRD, G-LRD, J-LRD). These metrics are coupled with explanations of how speedups are achieved by ensuring that the additional FLOPs remain manageable within GPU compute capabilities.

3. End-to-End Benchmark. Beyond the micro-benchmarking of attention score computation, our paper also incorporates broader end-to-end decoding latency benchmarks. These benchmarks highlight Palu’s overall decoding efficiency improvements in practical usage scenarios (Sec 4.5 and Figure 5).

By including a broad spectrum of sequence lengths, decomposition methods, and practical use cases, we ensure that the findings are not cherry-picked at all but instead comprehensively demonstrate Palu's effectiveness in improving LLM decoding efficiency.

(Q1). The statement "However, this approach poses significant computational challenges during runtime that make it impractical for deployments" lacks experimental results and theoretical support

Ans: Thanks for the good suggestions. To provide additional clarity, we have benchmarked the time required for performing SVD directly on the KV-Cache in comparison to the time needed for a forward pass through a decoder block, as shown in the Table R2 below:

Table R2. Latency comparison between performing SVD on the KV-Cache and a single forward pass through a transformer decoder block, using the model configuration of Llama-2-7B.

These results highlight that performing SVD on the fly introduces significant computational overhead. Specifically, runtime SVD is approximately 5–10 times slower than a forward pass through the decoder block, making it infeasible for real-time inference.

Additionally, SVD is highly precision-sensitive and requires higher precision (e.g., fp32 or fp64), introducing further type casting overhead if the base model is executed in low precision (e.g., fp16). This sensitivity to precision adds to the computational burden. 

Given these findings, Palu adopts an alternative approach by performing weight matrix decomposition offline to yield low-rank projection matrices. With these decomposed low-rank matrices, Palu achieves KV-Cache compression by caching the intermediate latent representations, effectively bypassing the runtime overhead associated with online SVD.

(Q2, Q3). Lack of theoretical proof, such as bounded error analysis and analyzation.

Ans: Thank you for raising this point. We acknowledge that providing a formal theoretical proof, such as bounded error analysis would strengthen the paper. While we lack theoretical analysis, we have conducted extensive experiments, as you have recognized, to validate the robustness, accuracy and efficiency of Palu under various conditions.

(Q4). In Section 4, the relationship between "compression rate" and "Avg. Bits" is unclear. Additionally, definitions of metrics such as "compression rate" are not specified.

Ans: Thank you for the suggestion. To clarify, the term “compression rate” refers to the overall reduction in KV-Cache size achieved by applying low-rank compression alone or in combination with quantization compared to the uncompressed baseline. “Avg. Bits” indicates the average number of bits required to store each element in the KV-Cache, considering quantization parameters, such as shared scales, that apply to a group of elements. We have updated Section 4 to be more clear.

(Q5). The code is not open-sourced

Ans: Thanks for your recommendation. We have uploaded the code via the OpenReview platform for your review. Please refer to the supplementary materials for further details.

Dear Reviewer 2i3G,

We sincerely appreciate your time and support for our work in the initial review—it truly means a great deal to us!

As a gentle reminder, the rebuttal cycle will conclude next Tuesday. We would be grateful if you could take a moment to review the additional analysis and clarifications we have provided in response to your comments. Additionally, the open-sourced code has now been uploaded. We would also like to confirm whether these updates have adequately addressed your concerns.

If you have any further feedback or suggestions, please don’t hesitate to let us know. Thank you again for your valuable time and effort!

Best regards,

Authors

Thank you very much for your thoughtful and constructive feedback on our work. We greatly appreciate the time and effort you have taken to provide your insights.

We want to follow up regarding your comments on Q2 and Q3. To address your concerns, we have made the following additions to our revised manuscript:

1. In Appendix L, we have included proof to demonstrate that the error on the KV-Cache induced by performing low-rank decomposition can be bounded.
2. In Appendix M, we have provided a detailed derivation and summary of the computing and memory complexity required to compute the attention score when applying the proposed Palu.

We hope this additional information can help address your concerns, and we are grateful for your valuable suggestions that have helped improve the rigor and clarity of our work.

Best regards,

Palu Authors

I sincerely thank the authors for their persistent responses. I will raise AC be the final judge on the responses.

Dear Reviewer 2i3G,

Thank you for your reply and detailed review of our paper. We truly appreciate your positive feedback and the rating. We also welcome any further questions or concerns you may have during the remaining two days of the discussion period.

(W1) Inefficiency with Positional Encoding.

Ans: While reconstruction might initially appear to introduce overhead, our proposed techniques—such as G-LRD and a co-designed GPU kernel—enable Palu to achieve a notable 1.89x speedup over standard attention by effectively balancing memory footprint and compute. For a more detailed explanation, please refer to the response in the global rebuttal.

(W2) Performance on Complex Tasks.

Ans:  We acknowledge that Palu-50% does not outperform lower-compression 2-bit quantized KV-Cache methods on complex tasks such as Code and Synthetic. However, quantization and hidden-dimension reduction are fundamentally distinct techniques with different strengths. Rather than comparing them head-to-head, we believe their complementarity is more important.

To illustrate this, we incorporated quantization-friendly enhancements into Palu’s low-rank compression. For instance, on coding tasks, Palu-30% combined with simple per-token quantization achieves comparable accuracy while delivering an additional 30% KV-Cache reduction (as shown in Table 3). Moreover, Appendix E demonstrates that Palu is compatible with Parameter Efficient Fine-Tuning (PEFT) methods like LoRA, which can further recover accuracy. These results underscore Palu’s flexibility and effectiveness in combining with other approaches to tackle challenging scenarios.

Dear Reviewer Fsn9,

We sincerely appreciate your time and thoughtful feedback on our work! We are glad to see that you lean toward acceptance of our submission—it truly means a great deal to us.

As a gentle reminder, the rebuttal cycle will conclude next Tuesday. We would be grateful if you could take a moment to review the additional experimental results and detailed explanations we have provided, as well as the global response. We also want to confirm whether our responses have adequately addressed your concerns.

Please feel free to let us know if you have any additional feedback or points of clarification—we would be happy to address them. Thank you again for your valuable time and effort.

Best regards,

Authors

Thank you for your thoughtful review and for taking the time to evaluate our rebuttal and updated paper. We are delighted to hear that you find our work valuable and appreciate your decision to raise the score. Below, we provide responses to your follow-up question:

Palu's performance on newer LLMs, such as Qwen2.5, which employ significantly compressed KV head states.

We appreciate your insightful suggestions and for raising concerns regarding Palu’s performance on LLM architectures with varying KV-head configurations. Table R7 provides experimental results on Qwen2.5-7B (4 KV heads) and Qwen2.5-1.5B (2 KV heads), showcasing Palu’s adaptability to architectures with compact KV-Caches.

Table R7. Zero-shot accuracy of Palu at different compression rates on Qwen2.5 models

Seeing from Table R7, Palu demonstrates strong performance on Qwen2.5-7B, a model variant with optimized KV states featuring only 4 KV-heads. Despite its compact KV-Cache design, which inherently makes compression more challenging, Palu achieves near-lossless accuracy at a 30% compression rate, incurring only a 0.48% average drop across six datasets. Even at a more aggressive 50% compression rate, Palu maintains robustness, with only a 2.2% average accuracy drop. These results confirm Palu’s adaptability and effectiveness in compressing models with fewer KV-heads, demonstrating its capability to handle compact KV-Caches while preserving high performance across tasks.

For Qwen2.5-1.5B, which possesses only 2 KV-heads, we observe that fully preserving performance with Palu becomes more challenging, particularly at an aggressive 50% compression rate. This phenomenon can be attributed to fewer KV-heads, which naturally results in smaller KV-Caches with higher information density, leaving less redundancy for compression techniques to exploit. However, our experiment shows that Palu still achieves robust accuracy at a competitive 30% compression rate, preserving a significant portion of the baseline performance across tasks with only 1.97% accuracy degradation on average. This highlights Palu’s ability to handle demanding scenarios while still delivering meaningful compression benefits, even for architectures with ultra-compact KV-Caches.

All in all, these findings demonstrate Palu’s scalability and robustness, effectively showing that it can identify and exploit redundancy even in the model architecture that KV-Cache have already been highly optimized.

We hope these additional experiments address your concerns regarding the robustness of the proposed Palu framework.

Thanks for your response. Below, we respond to your follow-up question:

Question: Is there no difference between joint-head and grouped-head approaches in 2 KV-head models?

Ans: Yes, you are correct. In the specific case of a 2-head model, the grouped-head approach is equivalent to the joint-head approach. This equivalence arises because grouping in G-LRD encompasses all available heads into a single group ($n_g = 1$).

As the reconstruction overhead for keys scales with the number of heads, the cost is significantly reduced in scenarios with extremely few heads. Consequently, Palu can leverage this reduced memory footprint by low-rank compression to alleviate memory bandwidth constraints and translate it into latency improvements in memory-bound large language model (LLM) decoding.

(W1) SVD for rank reduction of parameter matrices, is not novel, as it has been employed in various studies (e.g., LASER[1]). Consequently, the corresponding optimization of KV cache storage is straightforward.

Ans: Thank you for highlighting the role of SVD in prior work for parameter matrix reduction. SVD has been utilized in studies like LASER for compressing parameter matrices; however, LASER did not consider the fundamental shift required to optimize KV-Cache storage through rank reduction on parameter matrices. Palu introduces a redefined caching strategy by replacing full-dimensional KV tensors with low-rank latent representations. Beyond this innovation, Palu contributes new decomposition schemes, custom GPU kernels, and quantization-friendly enhancements—none of which are implemented or required in LASER. The global response above provides a detailed explanation of Palu’s key contributions and the challenges we addressed to deliver an efficient and practical solution.

[1] The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction. https://arxiv.org/abs/2312.13558

(W2) The paper does not provide open-source code

Ans: Thank you for raising this concern. We will disclose our open-source implementation upon acceptance of the paper. In the meantime, we have uploaded the code via the OpenReview platform for your review. Please refer to the supplementary materials for further details.

(W3) Lack of discussion with Token-level compression methods

Ans: Thank you for bringing up this concern. We would like to first clarify that Palu focuses on hidden-dimension compression, which operates in a fundamentally orthogonal direction to token-level methods. Token-level compression techniques, such as H2O, optimize KV-Cache efficiency by identifying and discarding less significant tokens. In contrast, Palu compresses the hidden dimensions across all tokens in a sequence using a layer-specific compression rate. As such, these methods are not directly comparable but could potentially be integrated to further optimize KV-Cache efficiency. 

To demonstrate the complementarity of our approach, we conducted additional experiments by integrating Palu with SnapKV [1], a more recent and advanced token-sparsity method than H2O. The results can be found in Table 1. As Table 1 shows, Palu-30% combined with SnapKV achieves performance comparable to applying each method independently in 5 out of 6 subjects (14 out of 16 tasks), with only a minor accuracy drop (~6.85%) on Synthetic datasets. These results provide strong evidence that token-sparsity methods and hidden-dimension reduction techniques can effectively complement each other.

Table R1. LongBench evaluation for Mistral-7B-v0.2 using the Token-Level Method, Palu, and the Combination of Both.

Additionally, we have expanded our discussion to cover the token-level compression methods, including H2O and SnapKV, in Appendix J of the revised submission. This provides a broader perspective on related work and situates Palu within the context of KV-Cache optimization research. We hope this helps to address your concern and clarifies the distinct focus and contributions of our approach.

Note: We acknowledge the accuracy drop on Synthetic datasets and believe it could be mitigated by adopting strategies inspired by token-level methods, such as preserving important tokens during compression. However, such advanced enhancements extend beyond the scope of this paper, which focuses specifically on the compression of the hidden dimension. We consider exploring these synergies as our future work.

(S1) Consider adding a mathematical derivation section.

Ans: We acknowledge that providing theoretical proof, such as bounded error analysis, would further strengthen the paper. Thanks for your suggestion. We will consider adding a mathematical derivation section to future work.

Dear Reviewer 12bo,

We sincerely appreciate your time and constructive feedback on our work!

We would like to kindly remind you that the rebuttal cycle will conclude next Tuesday. It would mean a great deal to us if you could take a moment to review the updated experimental results and detailed explanations we have provided, particularly addressing your concerns regarding the unique focus of Palu compared to related work and the associated discussion.

Your support is vital to our work, and we would be immensely grateful for your consideration and feedback.

Thank you again for your valuable time and effort.

Best regards,

Authors

Dear Reviewer 12bo,

We would like to kindly follow up regarding our recent revision. In line with your suggestion, we have included a theoretical proof in Appendix L demonstrating that the error on the KV-Cache induced by the rank reduction of parameter matrices can be bounded.

Once again, we sincerely thank you for your thoughtful feedback and the time you have dedicated to improving our submission.

Sincerely,

Palu Authors

Dear Reviewer 12bo,

We hope we have addressed all your concerns. As the discussion period concludes on Tuesday, December 2, we wanted to follow up to see if you have any additional questions regarding our previous responses, clarifications, or supporting experiments. If there are specific aspects you feel require further explanation, we would greatly appreciate it if you could let us know so that we can address them promptly.

We sincerely welcome any further questions or feedback you may have as we approach the end of the discussion period.

Thank you once again for your time, thoughtful feedback, and the effort you have dedicated to reviewing our work.

Sincerely,

Palu Authors