AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees

We sincerely thank you for your thorough and insightful review. Below, we address your concerns point by point. Meanwhile, we would like to respectfully clarify some potential misunderstandings raised in the listed weaknesses:

---

Q1. “Extra training complexity... makes the method less plug-and-play than purely retrieval-based or hard-pruning approaches.”

Response:

We think that “train-free” and “plug-and-play” are not fully equivalent concepts. Plug-and-play emphasizes the ability to integrate a module seamlessly into existing LLMs, without relying on their internal architecture or altering the original model. This is exactly what our AdmTree achieves, it can be adapted to any Transformer-based LLM without structural modifications.

Furthermore, although AdmTree requires training, the cost is orders of magnitude smaller than training an LLM from scratch to handle long contexts. We believe this offers a practical trade-off between efficiency and performance.

On the other hand, while retrieval-based methods require no training, their performance is consistently inferior to our AdmTree. As suggested by Reviewer eT7L, we also explored more advanced RAG variants such as Order-Preserving (OP) RAG. As shown below, it still brings marginal improvements. Additionally, we observed that RAG can even have a negative impact on summarization and some other tasks, highlighting its limited generalization across task types. This issue has also been mentioned in recent GraphRAG [1], which aims to improve retrieval coverage for tasks that require global information.

Additionally, training a powerful embedding model for RAG also incurs considerable cost, especially for nowadays LLM-based embedding models.

[1] From local to global: A graph rag approach to query-focused summarization. https://arxiv.org/pdf/2404.16130?

---

Q2. “There may be a mismatch between the compressed input structure and the model’s inductive biases.”

Response:

Thank you for raising this concern. Our goal is to keep the original LLM’s semantic space intact, and instead align the semantic space of the newly introduced gist tokens to that of the frozen LLM. This ensures that AdmTree is model-agnostic, plug-and-play, and requires much less training data.

If the LLM’s parameters are activated and involved in training, it would require an enormous scale of training data to align the two semantic spaces while preserving the LLM’s original semantic understanding. This may contradict the fundamental goal of context compression and therefore is not a practical solution. Also, we previously tried training both the original LLM and the new parameters in preliminary experiments, but found that it failed to converge.

In fact, many representative prior works (e.g., Gisting, ICAE, Activation Beacon) similarly keeps the backbone frozen and trains only the newly introduced parameters. In the future, we will consider training strategies inspired by multimodal LLMs, using multi-stage training with more data and separating training of different modules at different stages. Thank you for your thoughtful concern.

---

Q3. “Why use separate projection matrices for gist tokens and regular text tokens in self-attention?”

Response:

As we mentioned in Q2, we want the AdmTree to gain compression ability without disrupting the LLM’s original semantic space. This design choice enhances plug-and-play applicability and significantly reduces alignment overhead.

---

Q4. “Have you considered combining AdmTree with KV cache compression (e.g., SnapKV)?”

Response:

Yes, we agree that combining AdmTree with KV cache compression methods is a promising step toward a more comprehensive and efficient solution.

This idea aligns well with the tree-node retrieval strategy proposed in Table 3 of our paper, which aims to reduce inference load. Based on your suggestion, we conducted an experiment by appending SnapKV (compression ratio 2×) to our method during decoding. The results are shown below:

While some performance degradation is observed due to cumulative compression, the hybrid system still outperforms baselines such as AutoCompressor and ICAE from Table 1. We believe that building a unified framework that incorporates different types of compression is a valuable research direction, and we thank you for this inspiration.

---

Q5. “In Equation 2, how is the entropy defined over a sequence of tokens X_i?”

Response:

For a sequence $X_i = [x_1, ..., x_n]$, the entropy is calculated as the average token-level entropy:

$\text{Entropy}(X_i) = \frac{1}{n} \sum_{t=1}^n \left( -\sum_{v \in \mathcal{V}} P(v \mid x_{<t}) \log P(v \mid x_{<t}) \right)$

We will clarify this in the revised version.

---

Q6. “The method uses standard next-token prediction loss rather than a compression-specific objective (e.g., preserving downstream task performance or semantic similarity).”

Response:

We employ the standard next-token prediction loss in both the pretraining and instruction-tuning stages. During instruction tuning, the task instruction (query) is not a target for prediction. Although the loss is not specifically designed for compression, it is still compression-aware, since the core goal is to train the attention branches of the gist tokens to compress the context semantic and align with the LLM’s original semantic space. Notably, the instruction-tuning phase encourages the AdmTree to maintain the same output for the task, i.e., preserves downstream task performance.

We fully agree with your suggestion on designing compression-specific objectives. For example, introducing a reconstruction or paraphrasing task, where the model learns to regenerate the original context based on gist tokens, could encourage more comprehensive semantic compression. We believe this is a promising direction and leave it for future exploration. Thank you again for your thoughtful suggestion.

---

Typos will also be corrected immediately. We hope our clarifications resolve your concerns and would greatly appreciate it if you would consider a higher score.

We sincerely appreciate your recognition of our AdmTree's design and the inspiration your comments brought us. Below, we address the concerns you raised in detail:

---

Q1. “The adaptive token budget is decided solely by an entropy-weighted perplexity score. Have the authors considered alternative signals like TF-IDF, mutual information, or a learned ranker?”

Response:

In our early experiments, we explored several alternative scoring strategies, including attention-based scores and pure perplexity-based ranking, but found them less effective and stable. We have now included additional results comparing different scores:

Empirically, our entropy-adjusted PPL shows better alignment with actual information density, since it simultaneously accounts for the difficulty of the sentences and the contextual uncertainty.

Besides, as you suggested, designing a lightweight learnable scorer is indeed a promising direction. For instance, such a scorer could jointly consider the input and the LLM’s own capacity, enabling it to dynamically assign scores that reflect how different LLMs truely perceive the information density of the same sentence.

---

Q2. “All internal nodes are compressed by a single self-attention layer followed by pooling. Can such a shallow module still propagate fine-grained cues in deep hierarchies?”

Response:

The primary function of the aggregator is to propagate semantic information from child nodes in a bottom-up manner. Since it does not need to maintain its own semantic representation, we opted for a lightweight single-layer Transformer to ensure training efficiency.

To further investigate how aggregator depth affects performance at different tree depths, we compared 1-layer and 3-layer aggregators under varying compression ratios (i.e., different tree depths), using Qwen-2-7B as the base LLM. We selected QA and summarization tasks from LongBench, as they require semantic of different levels. LongBench allows inputs up to 32K tokens, so under 2× compression, tree depths can easily exceed 10 layers. The experimental results are as follows:

We observe that deeper aggregators do indeed improve performance, especially in summarization tasks where higher-level semantics are more critical. This suggests that the current aggregation design still has room for improvement. Unlike our current shared lightweight aggregators, future work could explore depth-aware or compression-aware aggregators. For example, under high compression ratios, leaf nodes must carry more semantic load, whereas in low-compression ratios, deeper trees shift the burden upward, making it essential for aggregators to appropriately balance fine-grained and global semantics.

---

Q3. “Results cover two 7B chat models. How does AdmTree perform on code/math tasks or other model sizes?”

Response:

The LongBench benchmark already includes a Code Completion task. In addition, we provided results on LongBench v2 and GSM8K in Tables 4 and 5 of the appendix. LongBench v2 requires long-context reasoning, while GSM8K is a widely used benchmark for mathematical reasoning.

To further demonstrate generalization across model sizes, we extended our experiments to include larger LLMs such as Qwen‑2.5‑7B and Qwen‑2.5‑14B. The results are shown below:

These results confirm that AdmTree maintains strong performance on larger and more advanced models, consistently outperforming the state-of-the-art method Beacon and the strong baseline Original LLM-FT.

---

We sincerely thank you again for your thoughtful suggestions and hope our clarifications address your concerns. We would greatly appreciate it if you would consider a higher score.

Thank you for your thoughtful evaluation and detailed comments. We address each concern below:

---

Q1. “The models used for evaluation (LLaMA‑2 and Qwen‑2) are relatively outdated compared to more recent LLMs.”

Response:

As explained in the paper, we use LLaMA‑2‑7B‑Chat because three out of the four compression baselines are based on it, ensuring fair comparisons. Our approach is model-agnostic and compatible with any Transformer-based LLM. To further validate its generality, we have extended our experiments to include more advanced and larger models, including Qwen‑2.5‑7B and Qwen‑2.5‑14B. The results are shown in the table below:

These results confirm that AdmTree continues to perform effectively on larger and more capable models, consistently outperforming the state-of-the-art method Beacon and the strong baseline Original LLM‑FT.

---

Q2. “The RAG scores for QA tasks appear to be quite low.”

Response:

Thank you for highlighting the pertinent RAG literature. Our RAG setup follows the original LongBench protocol. We use a chunk size of 500 and retrieve the top‑k most query‑similar chunks to fill the model’s context window as fully as possible, without preserving the original chunk order.

Compared to the paper you mentioned, the relatively low RAG scores in our work mainly stem from two factors. First, we use smaller open-source models, while the cited works rely on much larger and more capable closed-source LLMs such as Gemini‑1.5‑Pro and GPT‑4o. Second, we evaluate on LongBench, while those works use InfiniteBench. These two benchmarks differ substantially and cannot be directly compared. Besides, the original LongBench paper also observed cases where RAG negatively impacted performance.

As shown below, we further explored the effectiveness of the Order-Preserving (OP) RAG method from the paper [1] you referenced, testing it with various embedding models.

While OP‑RAG led to modest improvements in some cases, it still failed to outperform AdmTree. Additionally, we observed that RAG offers negative impact in summarization tasks, underscoring its limited generalization across task types. In contrast, AdmTree demonstrates stronger and more consistent generalization across a diverse set of scenarios.

---

Q3. “What are the detailed configurations for the RAG experiments?”

Response:

Please refer to the response to Q2.

---

Q4. “Why does the original LLM exhibit a speed-up in Table 1 for LLaMA‑2‑7B?”

Response:

The original LLaMA‑2‑7B model has a context window limit of 4096 tokens. Through fine-tuning (Original LLM‑FT), we extended the model’s window length to accommodate the full input from LongBench without truncation. As a result, the original model runs with shorter inputs and thus achieves significantly lower latency compared to the extended model, although its performance is naturally weaker.

---

Q5. “What compression ratio was used in Figure 4?”

Response:

In Figure 4, for each context length, we selected the smallest compression ratio from the set {2, 4, 8, 16, 32} to ensure that (context_length / compression_ratio) < model_context_window. We will clarify this in Section 5.3 of the final version.

---

Q6. “For the needle-in-a-haystack evaluations, what types of noise and needle content were used? Was the noise content repetitive?”

Response:

We followed the original Needle in a Haystack setup. The noise content comprises essays by Paul Graham, with no repetition. The needle is a synthetic question (to prevent real-world data leakage) about San Francisco.

---

Q7. “Limitations were missing from conclusion.”

Response:

Thank you for catching this oversight. We will reintroduce the following limitations in the Conclusion section: (1) Our current experiments are focused on English long-text benchmarks, and the multilingual or cross-lingual capabilities of AdmTree remain unexplored. (2) AdmTree performs compression only on the input side. It does not currently address the compression of long outputs, which is increasingly important as LLMs produce longer intermediate reasoning chains.

---

We sincerely thank you again for your constructive feedback. We hope that our responses have addressed your concerns and support a higher score!

We sincerely thank you for the detailed and thoughtful review, as well as the recognition of our work. We address your comments and questions below:

---

Q1. On causal structure and use of self-attention from subsequent segments

“AdmTree utilizes self-attention from subsequent part of texts, mitigating the unidirectional limitation of causal LLM... it could be somewhat cheating for predicting the next word.”

Response:

Thank you for pointing out this subtle but important concern. To clarify, AdmTree preserves the causal structure required for next-token prediction. The self-attention involved in tree aggregation (Section 3.3) operates outside the autoregressive decoding path. It only summarizes the already-seen gist tokens to build semantic tree. During next-token prediction in segment $s_k$, the model only attends to tokens from $s_k$ (autoregressively, via causal masking), and the semantic tree $T_{<k}$, constructed solely from previous sub-segments. Thus, no future tokens beyond position $j$ (the predicted token) are accessed or leaked into the prediction path.

---

Q2. On the use of both perplexity and entropy in Equation (2)

“Perplexity and entropy are actually the same thing, just differing about some transformation.”

Response:

Perplexity and entropy would be mathematically equivalent only when entropy were computed as the cross-entropy loss on the target token. But in our framework, they serve different roles. For segment $X_i = [x_1, \dots, x_n]$, PPL measures the model's sequence-level prediction difficulty, which is defined as: $\text{PPL}(X_i) = \exp\left(-\frac{1}{n} \sum_{t=1}^{n} \log P(x_t \mid x_{<t})\right)$ This reflects how “surprised” the model is by the segment. In contrast, Entropy in our design is the average token-level entropy, which is calculated as: $\text{Entropy}(X_i) = \frac{1}{n} \sum_{t=1}^{n} \left(-\sum_{v \in \mathcal{V}} P(v \mid x_{<t}) \log P(v \mid x_{<t})\right)$ where $\mathcal{V}$ is the LLM vocabulary. This captures the intrinsic contextual complexity at each position. High PPL may arise from genuine surprising content or from low-value noise. The entropy term helps suppress segments that are surprising but semantically flat, avoiding over-allocation of gist tokens to noise. Empirically, our entropy-adjusted PPL shows better alignment with actual information density, especially in heterogeneous contexts.

We also conducted comparative experiments showing the superiority of our entropy‑adjusted PPL metric:

---

Q3. On flattening the hierarchical structure before feeding into flat LLMs

Response:

“It is a bit pity that the hierarchically organized gists are just flattened... If LLM itself utilizes hierarchical structure, flattening might be unnecessary.”

This is an insightful comment and aligns with our future vision! We currently flatten the hierarchical gist tree to maintain compatibility with Transformer-based architectures, but this is a practical constraint, not a conceptual necessity.

However, emerging architectures like Hierarchical Reasoning Model [1], which employs separate recurrent modules for planning and execution while using very few parameters, perform deep hierarchical reasoning without any flattening and still achieve state-of-the-art results. Similarly, recent research into diffusion-based reasoning, such as Diffusion-of-Thought (DoT) [2], allows reasoning steps to form non-linear, chain-of-thought-like trajectories during diffusion rather than a strict left‑to‑right generation.

Together, these models illustrate that hierarchical representations can be processed natively. In future work, AdmTree’s semantic tree could serve as a compression front-end feeding into these hierarchy-native backbones, thereby eliminating the need for flattening and enabling more natural hierarchical reasoning. We will add this forward-looking discussion in our conclusion section.

[1] Hierarchical Reasoning Model. https://github.com/sapientinc/HRM

[2] Diffusion of thought: Chain-of-thought reasoning in diffusion language models. NIPS'24.

---

Thank you for your valuable insights and kind recognition. Your feedback has inspired us and made our work better!