Response to Reviewer 7eDi

We sincerely thank you for your valuable and insightful comments, which have greatly helped us improve our manuscript. Below, we provide detailed responses to each issue you raised.

W1. Although Group Merging and LSA are proposed, there is a lack of in-depth analysis to explain why this design can maintain semantic integrity and how to determine the optimal group size?

Thank you for pointing this out. We clarify and further explain this design as follows:

• Semantic Integrity: Group Merging maintains semantic integrity primarily due to uniform token aggregation within groups. Unlike traditional methods that unevenly emphasize certain "anchor" tokens, our method equally aggregates all tokens, effectively preserving diverse semantic nuances. Layer Semantic Alignment (LSA) module further ensures semantic integrity by aligning higher-layer semantic vectors to lower-layer semantic spaces using Transformer blocks initialized with decoder parameters, thus bridging the inherent semantic gaps.
• Optimal Group Size: We selected the group size based on practical compression requirements and empirically validated through ablation studies (Appendix B and Table 2). Our experiments revealed stable performance across multiple group sizes (4x, 8x compression). We recognize your valuable suggestion and will add adaptive group-size optimization based on text characteristics to our future work.

W2. The experiments were mainly conducted on the 7B model and lacked verification on larger-scale models (such as 13B and 70B), which limits the demonstration of the universality of the method.

Thanks for highlighting this point. We have now conducted additional experiments using LLaMA-2-13B. For simplicity, the performance of method under Nx compression rate constraints will be denoted as 'Nx method' in the following.

NQ HQA MQA (LLaMA-2-13B as backbone)

We present these results clearly in the revised version, demonstrating that GMSA's performance gains persist across larger-scale models, indicating strong universality. We plan to conduct further experiments with 70B models when more resources are available.

W3. The experiments mainly focus on QA tasks and lack verification ... which limits the versatility of the method.

We acknowledge this limitation. To directly address your suggestion, we have conducted additional experiments to evaluate GMSA on summarization (CNN / DailyMail [1]), code completion (Repobench [2]), math capability (GSM8K [3]) a wide range of disciplines (MMLU [4]). We compare GMSA with strong baseline, i.e., Activation Beacon [5].

We trained and evaluated GMSA on the training and test sets of the CNN / DailyMail dataset, respectively. For the Repobench dataset, we split it into training and test sets at a ratio of 4:1 and conducted training and evaluation accordingly.

CNN / DailyMail (LLaMA-2-7B as backbone)

Repobench (Qwen-2-7B as backbone)

We directly evaluate the performance of the GMSA version trained on the mixed NQ, HQA, and MQA dataset on MMLU and GSM8K to assess its short-context capability and generalization.

MMLU

GSM8K

GMSA even outperforms the untrained LLaMA-2-7B on MMLU, demonstrating the superior short-context capability and generalization of our method. However, its performance slightly drops on GSM8K, likely because GMSA was not trained on mathematical data and thus shows somewhat reduced performance in this specific domain.

These new experiments demonstrate that GMSA maintains robust performance across broader generation-centric scenarios.

W4. Group Merging essentially destroys the contextual continuity of natural language ... may be segmented into different groups.

We recognize the reviewer's concern about potential disruption of continuity. While Group Merging does unavoidably segment natural-language sentences, the generated soft tokens remains continuous in the dense vector space because the relative order of every compressed group mirrors that of the original text. Empirical evidence (Table 2 and the case study in Appendix H) confirms that semantic continuity and integrity are preserved at a high level after merging.

W5. The paper claims that "equally considers each group" is an advantage ... treatment may actually harm the retention of key information.

Thank you for the insightful question. As Figure 1 illustrates, conventional compression paradigms suffer from uneven semantic learning. GMSA adapts to downstream tasks through a two-stage training process. In the first stage (i.e., Autoencoder Training) the sole objective is to preserve the complete semantics, so treating every group equally is advantageous; this uniformity enables the learning of soft tokens that faithfully encode the full semantic content. The second stage (i.e., Knowledge Extraction Fine-Tuning (KEFT))then focuses on teaching the decoder how to extract and utilize the question-relevant key information contained within those soft tokens.

Thank you for the insightful question. As shown in Figure 1, traditional compression paradigm suffers from uneven semantic learning. GMSA addresses this through a two-stage training procedure. In the first stage (Autoencoder Training), the sole objective is to preserve the semantics of every original token as completely as possible via soft tokens. Hence, treating each group equally is advantageous; this uniformity enables the model to learn soft tokens that carry the full semantic content, rather than over-emphasizing certain key information (which would dilute the semantics of other tokens and hinder complete preservation). The second stage (Knowledge Extraction Fine-Tuning, KEFT) then focuses on teaching the decoder to extract and leverage the question-relevant key information embedded in those soft tokens.

W6. A fixed group size treats all texts equally, but different types of text ... how to adaptively adjust the grouping strategy based on content characteristics.

We appreciate the reviewer highlighting this important perspective. In this work, we employed a fixed group size to maintain architectural simplicity and computational efficiency, and our experiments demonstrate that this approach works well across a variety of datasets and domains (e.g., QA on Wikipedia-based passages, multi-hop reasoning, etc.). That said, we fully agree that different text genres may benefit from dynamic or adaptive grouping strategies.

As part of our future work, we plan to explore content-aware adaptive grouping, possibly by using shallow classifiers or linguistic heuristics to estimate information density or discourse structure. These adaptive mechanisms could guide more semantically-aligned grouping decisions based on textual features, potentially improving both compression fidelity and downstream performance. We thank the reviewer for encouraging this promising direction.

W7. The experiments in the paper mainly focus on the final QA performance, but no special experiments are designed to verify whether Group Merging really retains the "complete semantics". For example, there is a lack of semantic similarity analysis and information theory analysis of the compressed representation.

We thank the reviewer for emphasizing the need for deeper validation of semantic retention. In fact, our experiments already include dedicated evaluations of completeness (Figure 4 and Appendix H):

• Text reconstruction test (Figure 4). We reconstruct the original context from the soft tokens generated by GMSA-AE and compute both token-matching metrics (e.g., BLEU, ROUGE and Prefix Exact Match) and semantic-matching metrics (e.g., BERT Score F1). The results show high performance on both metrics, and GMSA consistently outperforms the Traditional Compression Paradigm AutoEncoder (TCP-AE) by at least 20 % across different sequence lengths and compression rates.
• Perplexity-based information analysis. We measure the perplexity of the reconstructed text from the soft tokens. GMSA-AE yields consistently lower perplexity than TCP-AE and, in some cases, even slightly lower than the original context itself (see Table 5 in Appendix H). This strongly supports the claim that Group Merging preserves complete semantics.

These experiments directly address the reviewer's concern and confirm the reliability of our design in retaining semantic integrity.

References

[1] See, Abigail, Peter J. Liu, and Christopher D. Manning. "Get To The Point: Summarization with Pointer-Generator Networks." ACL (1), Association for Computational Linguistics, 2017, pp. 1073-1083.

[2] Liu, Tianyang, Canwen Xu, and Julian J. McAuley. "RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems." ICLR, 2024.

[3] Cobbe, Karl, Vineet Kosaraju, Mohammad Bavarian, et al. "Training Verifiers to Solve Math Word Problems." CoRR, vol. abs/2110.14168, 2021.

[4] Hendrycks, Dan, Collin Burns, Steven Basart, et al. "Measuring Massive Multitask Language Understanding." ICLR, 2021.

[5] Zhang, Peitian, et al. "Long Context Compression with Activation Beacon." ICLR, 2025.

Dear Reviewer 7eDi,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript and providing constructive feedback. Your insightful comments have significantly helped us improve the manuscript.

We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you. Thank you very much for your time and consideration. We are looking forward to hearing from you.

Best regards,
The Authors

Response to Reviewer NUaJ

We sincerely thank you for your valuable and insightful comments, which have greatly helped us improve our manuscript. Below, we provide detailed responses to each issue you raised.

W1 & Q1. Although this paper studies the context compression, it is not clear that whether the proposed approach would reduce the standard LLM metrics (e.g. MMLU, GSM8k etc)

Thank you for highlighting this important point. Our primary experiments focused on two core dimensions:

1. Context Restoration, to evaluate the method's ability to precisely memorize and reconstruct original context.
2. Question Answering tasks, to assess the downstream knowledge extraction capability from compressed representations.

We agree it is valuable to understand GMSA's influence on standard LLM metrics. We have subsequently conducted additional experiments on standard LLM benchmarks (MMLU and GSM8K) and will incorporate these results into the revised version. We compare GMSA with the strong baseline, i.e., Activation Beacon [3]. For simplicity, the performance of method under Nx compression rate constraints will be denoted as 'Nx method' in the following. Below are the preliminary results:

We directly evaluate the performance of the GMSA version trained on the mixed NQ, HQA, and MQA dataset on MMLU and GSM8K to assess its short-context capability and generalization.

MMLU

GMSA even outperforms the untrained LLaMA-2-7B on MMLU, demonstrating the superior short-context capability and generalization of our method. However, its performance slightly drops on GSM8K, likely because GMSA was not trained on mathematical data and thus shows somewhat reduced performance in this specific domain.

GSM8K

Initial evaluations show that GMSA maintains competitive performance compared to original LLaMA-2-7B, confirming the approach's generalizability without performance degradation on standard tasks.

W2 & Q4. Although this paper shows strong performance ... whether this approach would achieve better performance than the standard LLM. & What is the performance comparing to the original LLAMA and QWEN on the proposed benchmarks (metrics wise, not efficiency)?

Thank you for pointing out the need for direct performance comparisons with the original LLaMA and Qwen models. For example, as shown in Table 1, the reported "Original Prompt" results already reflect the performance of the original LLaMA2-7B. Furthermore, we conducted additional experiments, in which we performed full fine-tuning on the original prompts using the same mixed training set (denoted as "Full Fine-tune Original Prompt") to thoroughly compare the effectiveness of GMSA and the original input. The results are as follows:

NQ HQA MQA (LLaMA-2-7B as backbone, Accuracy as Metric)

The results demonstrate that GMSA outperforms Full fine-tune Original Prompt at an 8x compression rate, fully illustrating the superiority of our method. The fact that it can even surpass full fine-tuning may be because GMSA effectively compresses the long context into compact and information-dense soft tokens, making it easier for the model to extract key information that is directly relevant to the input questions.

Q2. For the autoencoder training stage ... information leak that causes the next token prediction task to be trivial?

During the autoencoder training, the decoder is tasked explicitly with context restoration conditioned on compressed representations (soft tokens). The decoder never sees the original raw token embeddings directly, only their compressed semantic summaries. Thus, information leakage is inherently controlled since the decoder does not have access to raw tokens but must reconstruct the original sequence from highly compressed representations.

Q3. I might miss some details here. I ... two training stages?

Thank you for the insightful question regarding whether two-stage training is necessary.

Theoretically, we can also formulate the two-stage training as a non-separate, end-to-end fine-tuning. In practice, however, this yields unstable loss curves and degraded performance. The instability arises from the inherent difference between the training objectives: Autoencoder Training forces the soft tokens to store the complete semantic representation, whereas Knowledge Extraction Fine-Tuning optimizes the decoder's capacity to retrieve and apply that stored knowledge. These divergent goals preclude a unified optimum under joint training.

Q5. If the proposed approach is useful, I wonder if this should be applied to all the LLM training paradigm?

We appreciate the suggestion regarding the potential applicability of our method to general LLM training paradigms. While our primary focus was context compression, our proposed methodology, i.e., Group Merging and Layer Semantic Alignment (LSA) module, can indeed serve as a general-purpose module within broader LLM frameworks. Exploring such extensions is an exciting future direction that we intend to pursue, especially given our initial promising results.

Q6. For the autoencoder training stage, I have one detail question. Let assume ... to fully consume all the tokens within the long context.

Thank you for the question. The LLM decoder only needs a short length; the exact value is determined by the predefined compression rate. When dealing with very long inputs, the decoder processes the text in a sliding-window fashion: after obtaining soft tokens from the first segment, these tokens are prepended to the next segment, and this process repeats sequentially.

References

[1] Hendrycks, Dan, Collin Burns, Steven Basart, et al. "Measuring Massive Multitask Language Understanding." ICLR, 2021.

[2] Cobbe, Karl, Vineet Kosaraju, Mohammad Bavarian, et al. "Training Verifiers to Solve Math Word Problems." CoRR, vol. abs/2110.14168, 2021.

[3] Zhang, Peitian, et al. "Long Context Compression with Activation Beacon." ICLR, 2025.

Response to Reviewer DgAD

We sincerely thank you for your valuable and insightful comments, which have greatly helped us improve our manuscript. Below, we provide detailed responses to each issue you raised.

W1. & Q1. Narrow task coverage ... remains untested. & Does GMSA help on generation-centric tasks (e.g., long-form summarization, dialogue, code completion) or non-retrieval settings?

We acknowledge this limitation. To directly address your suggestion, we have conducted additional experiments to evaluate GMSA on summarization (CNN / DailyMail [1]), code completion (Repobench [3], which max length is 16K), math capability (GSM8K [5]), and a wide range of disciplines (MMLU [4]). We compare GMSA with the strong baseline, i.e., Activation Beacon [6]. For simplicity, the performance of method under Nx compression rate constraints will be denoted as 'Nx method' in the following.

We trained and evaluated GMSA on the training and test sets of the CNN / DailyMail dataset, respectively. For the Repobench dataset, we split it into training and test sets at a ratio of 4:1 and conducted training and evaluation accordingly.

CNN / DailyMail (LLaMA-2-7B as backbone)

Repobench (Qwen-2-7B as backbone)

We directly evaluate the performance of the GMSA version trained on the mixed NQ, HQA, and MQA dataset on MMLU and GSM8K to assess its short-context capability and generalization.

MMLU

GSM8K

GMSA even outperforms the untrained LLaMA-2-7B on MMLU, demonstrating the superior short-context capability and generalization of our method. However, its performance slightly drops on GSM8K, likely because GMSA was not trained on mathematical data and thus shows somewhat reduced performance in this specific domain.

These new experiments demonstrate that GMSA maintains robust performance across broader generation-centric scenarios.

W2. Residual compute overhead. The full encoder ... less dramatic than some KV-cache pruning methods on very long inputs.

Thank you for highlighting this point. Although our encoder retains linear complexity with input length, it significantly reduces the actual inference latency (~2x speedup) by compressing context into fewer soft tokens, balancing efficiency and performance. While KV-cache pruning methods may achieve more dramatic FLOPs reduction for very long sequences, they sacrifice flexibility and semantic completeness, which GMSA preserves. We will clarify this trade-off explicitly in our revision.

We conduct experiment on the samples with the max length of 32K on NarrativeQA [2]:

NarrativeQA (We set max length to 32K, Qwen-2-7B as backbone)

These new experiments demonstrate that GMSA accelerates end-to-end generation even with the max length of 32k.

W3. Limited statistical rigor. Results are ... run-to-run variance.

Thank your for the good suggestion. Actually, we fixed the random seed at 42 and all packages' version for every experiment, ensuring full reproducibility. We agree and will include confidence intervals and statistical significance tests in our revised results to enhance rigor.

W4. No human evaluation. Fidelity of ... high compression is unknown.

We acknowledge this important feedback. Given resource constraints during initial experiments, human evaluations were omitted. In future work, we plan comprehensive user studies to assess the subjective fidelity of compressed prompts.

W5. Hyper-parameters hidden in appendix and code unreleased. ... hinders immediate reproducibility.

Thank you for highlighting this issue. We will move key hyper-parameters (LoRA ranks, learning rates, GPU hours) into the main body of the paper. Regarding the code, we apologize for the lack of clarity. Actually, the Supplementary Material (i.e., the .zip file) already contains the unpolished implementations of GMSA and all baselines. We will promptly clean up and open-source the code to guarantee full reproducibility as soon as possible.

W6. Incremental originality. While ... rather than a fundamentally new paradigm.

Group Merging targets uneven semantic learning by using average pooling to enforce more uniform learning; its approach to this problem is original. Layer Semantic Alignment (LSA) starts with the initial weights of a few early decoder layers, but its core motivation is to bridge the large gap between the high-level semantic space of the encoder and the low-level semantic space of the decoder, which is distinct from the decoder's own role, making it innovative. GMSA combines Group Merging and LSA module, achieving substantial empirical gains that demonstrate the method's superiority and effectiveness.

W7. Scalability unanswered. The paper does not explore compression ratios beyond 8x or contexts beyond 32K tokens, nor does it test on larger/smaller LLM backbones, leaving open how robust the approach is across scales.

Thank you for this suggestion. Although hardware limitations initially constrained our experiments, GMSA is designed to be scalable. To demonstrate this, we have conducted additional experiments at 32K token max length on NarrativeQA with 16x compression rate and use LLaMA-2-13B as backbone on three QA datasets, i.e., NQ, HQA and MQA:

NarrativeQA (We set max length to 32K, Qwen-2-7B as backbone)

NQ HQA MQA (LLaMA-2-13B as backbone)

The experimental results demonstrate that GMSA has excellent scalability.

Q2: Do the compressed prompts ever drop critical facts or cause answer hallucinations that automatic metrics miss?

Thank you for the question. As shown in Figure 4, we evaluate GMSA trained with Autoencoder Training (GMSA-AE) on text restoration tasks. Both token-matching metrics (e.g., BLEU Score) and semantic-matching metrics (e.g., BERT Score F1) are markedly higher (by at least 20%) than those of the conventional-compression AutoEncoder, demonstrating GMSA's strong semantic preservation. Regarding key information, NQ, HQA, and MQA require concise answers directly relevant to the question; GMSA substantially outperforms baselines on all three datasets, highlighting its superior retention of critical facts in compressed prompts. As for answer hallucinations, none appear in our examined cases, i.e., every response stays strictly related to the input.

NQ HQA MQA (LLaMA-2-13B as backbone, Accuracy as Metric)

Broader Impact Analysis

Thank you for emphasizing the importance of a broader impact analysis. We will include a dedicated section (≤ 1/2 page) in the revised version, explicitly discussing potential societal implications:

• Misuse Risks: Accelerated inference and prompt compression could facilitate large-scale generation of misinformation or disinformation campaigns.
• Privacy Concerns: Edge deployments enabled by our method might inadvertently process sensitive user data without proper safeguards.
• Bias and Fairness: Summarization through compression may inadvertently amplify existing biases by selectively preserving information.

We will outline the following mitigation strategies:

• Checkpoint Controls: Provide access-gated model checkpoints to prevent misuse.
• Misuse Monitoring: Publish clear guidelines and monitoring tools alongside our code release.
• Bias and Robustness Evaluations: Include protocols for evaluating biases and robustness explicitly in future releases.

Additionally, we will clarify positive impacts such as improved energy efficiency and broader accessibility to large models, fostering a balanced understanding of GMSA's societal implications.

References

[1] See, Abigail, Peter J. Liu, and Christopher D. Manning. "Get To The Point: Summarization with Pointer-Generator Networks." ACL (1), Association for Computational Linguistics, 2017, pp. 1073-1083.

[2] Kociský, Tomáš, Jonathan Schwarz, Phil Blunsom, et al. "The NarrativeQA Reading Comprehension Challenge." Transactions of the Association for Computational Linguistics, vol. 6, 2018, pp. 317-328.

[3] Liu, Tianyang, Canwen Xu, and Julian J. McAuley. "RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems." ICLR, 2024.

[4] Hendrycks, Dan, Collin Burns, Steven Basart, et al. "Measuring Massive Multitask Language Understanding." ICLR, 2021.

[5] Cobbe, Karl, Vineet Kosaraju, Mohammad Bavarian, et al. "Training Verifiers to Solve Math Word Problems." CoRR, vol. abs/2110.14168, 2021.

[6] Zhang, Peitian, et al. "Long Context Compression with Activation Beacon." ICLR, 2025.

Response to Reviewer Vink

We sincerely thank you for your valuable and insightful comments, which have greatly helped us improve our manuscript. Below, we provide detailed responses to each issue you raised.

W1. Limited Context Length Evaluation: My main concern of this work is that experiments for QA tasks are performed on contexts only up to ... The lack of results on longer sequences leaves the scalability of the method as an open question.

We acknowledge your valuable point regarding the evaluation context length. To directly address this concern, we have conducted additional experiments evaluating GMSA on NarrativeQA [1] (we set max length to 32K tokens), specifically in more diverse long-text scenarios such as code completion and summarization. For simplicity, the performance of method under Nx compression rate constraints will be denoted as 'Nx method' in the following. Below are the preliminary results:

NarrativeQA (We set max length to 32K, Qwen-2-7B as backbone)

These results confirm that GMSA is highly scalable and retains strong performance even with considerably longer contexts, further reinforcing the practical utility and effectiveness of our proposed framework.

In addition, we conducted experiments on LLaMA-2-13B to fully verify the scalability of our method in terms of model size:

NQ HQA MQA (LLaMA-2-13B as backbone, Accuracy as Metric)

The experimental results show that when the backbone is LLaMA-2-13B, the results still significantly outperform the Original Prompt, which proves the effectiveness and superiority of GMSA.

Moreover, we emphasize that the GMSA framework itself inherently supports arbitrary context lengths. The encoder module in GMSA is pluggable and length-agnostic. Replacing the current encoder (LLaMA-2 based) with a model supporting larger contexts, such as Qwen-1M or other long-context LLMs, allows GMSA to naturally scale to extremely long sequences (100k tokens or more).

We explicitly mentioned our initial hardware limitations and their implications in Appendix J. The unorganized versions of the GMSA and baseline codes are included in the Supplementary Material (i.e., the .zip file). To facilitate community validation and further exploration, we will organize and open-source our codes as soon as possible.

W2. Missing citations and comparisons (LLoCO). The work misses citation and comparison with prior work LLoCO, ... The authors should properly compare this work.

We thank the reviewer for pointing out this missing citation. We acknowledge the relevance of the work LLoCO [2], which indeed utilizes a similar encoder-decoder architecture and a decoder-only fine-tuning strategy for downstream tasks, analogous to our Knowledge Extraction Fine-tuning (KEFT).

We have now thoroughly reviewed LLoCO and compared it explicitly with our proposed GMSA approach.

Key Differences and Contributions:

• Group Merging: GMSA proposes a novel Group Merging strategy to evenly retain semantics, significantly mitigating uneven semantic learning observed in prior methods.
• Layer Semantic Alignment (LSA): GMSA introduces an explicit cross-layer semantic alignment module that addresses the large semantic gap between high-level abstract representations and low-level decoder inputs, an aspect that is not considered in LLoCO.
• Knowledge Extraction Fine-tuning (KEFT): KEFT selectively fine-tunes only the weight matrices $W_Q$, $W_K$, and $W_V$ in each decoder layer. This approach is based on the observation that the attention mechanism (Q/K/V) primarily coordinates information flow and enables context integration, while the feed-forward network (FFN) functions as a knowledge storage module, akin to a memory bank [3]. Therefore, fine-tuning only the attention projections is better to adapt the model for knowledge-intensive tasks. In contrast, LLoCo applies Low-Rank Adaptation (LoRA) to the entire decoder, not just the attention components.

We trained on the NQ dataset using the default settings of the LLoCO open-source code, and the test results are as follows:

NQ (LLaMA-2-7B as backone)

We will integrate a detailed citation and comparison of LLoCO in the revised version, clearly highlighting these differences and our method's unique contributions.

Q1. What is the base model used for GMSA? Is it LLaMA2-7B or Qwen2.5-7B?

We apologize for the lack of clarity. We confirm that the base model used for GMSA results in Table 1 is LLaMA-2-7B, ensuring a fair comparison with the listed baseline methods. We realize this clarification was unclear in our original manuscript and will explicitly state this in the revised version.

References

[1] Kociský, Tomáš, Jonathan Schwarz, Phil Blunsom, et al. "The NarrativeQA Reading Comprehension Challenge." Transactions of the Association for Computational Linguistics, vol. 6, 2018, pp. 317-328.

[2] Tan, Sijun, Xiuyu Li, and Shishir G. Patil. "LLoCO: Learning Long Contexts Offline." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2024, pp. 17605-17621.

[3] Geva, Mor, Roei Schuster, Jonathan Berant, et al. "Transformer Feed-Forward Layers Are Key-Value Memories." EMNLP (1), Association for Computational Linguistics, 2021, pp. 5484-5495.

[4] Zhang, Peitian, et al. "Long Context Compression with Activation Beacon." ICLR, 2025.