Language Models (Mostly) Know When to Stop Reading

Dear Reviewer 2h3k,

We sincerely thank you for your constructive feedback and recognition of our paper's strengths, including clear writing and comprehensive experiments and evaluations. We appreciate you taking the time to review our paper and would like to address your concerns as follows:

Technical Innovation and Core Contributions

Thank you for this important question! We would like to clarify that there are two orthogonal approaches to efficiency: (1) methods that approximate transformer computations during inference (like TokenSelect, which performs sparse attention by dropping low-attention tokens), and (2) methods that reduce input context length. These approaches are complementary, as any context compression method can benefit from computational approximations like KV optimization.

Our work falls in the second category, focusing on context reduction rather than computational approximation. Within this space, we make two novel contributions that distinguish our approach from existing context compression methods:

First, we are the first to leverage internal LLM representations to decide when to stop processing, discovering that certain attention heads naturally encode “sufficiency signals” that indicate when enough context has been seen. This is a previously unexplored aspect of transformer interpretability.

Second, rather than treating context compression as a separate preprocessing module (as in prior work [1]), we make it an inherent part of how the model encodes text during inference. This integration enables dynamic, model-driven decisions rather than fixed compression rules.

We compare against all relevant baselines within the context compression category. Regarding the Li et al. (2023) reference, we note this was the proof-of-concept version later developed into LLMLingua, and we do compare against the improved LLMLingua methods in our experiments.

Due to page limits, our discussion of the complementary nature of these two categories was moved to Appendix H.4 (Potential Combination with KV Cache Optimization). In the revision, we will expand the discussion in the main text and include both mentioned relevant work in the related work section to better position our contributions. Thank you again for raising this excellent question!

Hyperparameter Sensitivity and Selection

Thank you for this insightful question! Among our three hyperparameters, only the classification threshold τ impacts the efficiency-accuracy trade-off, while the other two (number of heads and ensemble size) are determined through standard hyperparameter search. Importantly, we use a single global threshold across all datasets (Tables 2 and 3), as our method does not require dataset-specific tuning. As shown in Figure 1, different threshold values create predictable efficiency-accuracy trade-offs, but the key finding is that a single global setting works effectively across all experimental conditions. This addresses the core concern about sensitivity - we don't need separate thresholds for each dataset type.

Practically, the threshold τ can be set once based on desired efficiency-accuracy preferences and deployed universally. We also provide detailed sensitivity analysis on number of heads and ensemble size in Appendix B.1 and H.1, showing stable performance across reasonable ranges. Additionally, for larger models, our self-prompting approach eliminates dependence on trained classifiers entirely, which highlights the potential extension of our method with a training-free alternative which does not require hyperparameter tuning.

Classifier Complexity and Obtainment

You raise an important question regarding the ease of obtaining classifiers and their applicability across model families. We clarify that training classifiers is straightforward, requiring only attention head activations and binary sufficiency labels. The training process follows a simple procedure: probe all attention heads, select top-performing heads, and train the classifiers. This one-time setup enables deployment across all tasks for that model. The classifiers are lightweight, using linear or tree-based architectures with minimal memory footprint (Appendix H.2). Binary sufficiency labels can be obtained from existing datasets when available (Appendix A.3), or generated using language model judges through prompting (Appendix A.4).

In terms of cross-model applicability, classifiers cannot be directly transferred across model families due to architectural differences in attention mechanisms and internal representations. This model-specific requirement aligns with previous interpretability work [2] and reflects a core research contribution: demonstrating that internal model representations encode sufficiency signals. While extracting these signals requires model-specific classifiers, our probing methodology itself is universal. Once trained, classifiers show consistent performance across tasks within the same model. This model-specific nature also reflects realistic differences in how various architectures process context, as different models exhibit different context utilization patterns (L220-222).

[1] Li, Yucheng, et al. "Compressing context to enhance inference efficiency of large language models." arXiv preprint arXiv:2310.06201 (2023).

[2] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

Once again, thank you for the thoughtful review and the opportunity to clarify these important aspects of our work. Your questions have helped us identify areas where we can strengthen our presentation and better highlight the novel contributions of our approach. Thank you for your valuable feedback and consideration of our work!

Best regards,

Authors

Dear Reviewer xgYF,

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of our method's strengths, including eliminating the need for pre-defined compression targets, insights into Transformer interpretability, and significant performance improvements. We would like to address your concerns and questions below:

Early cutoff when information spans across different parts of content & Solutions for early cutoff

You raise an important concern about early cutoff. We would like to highlight that our method is carefully designed to handle this scenario. Our dataset construction ensures gold information spans uniformly across contexts (Appendix A.2), specifically testing scenarios where information is distributed across different parts. Our multi-hop evaluation on HotpotQA, MUSIQUE, and Multi-hop Key-Value Retrieval requires integrating information from multiple locations throughout the context. Table 2 shows our method achieves 29% accuracy on multi-hop tasks, outperforming LLMLingua2 (27.8%) with 1.33× token reduction, demonstrating effectiveness when information spans across distant locations.

Additionally, we address early cutoff through multiple robust mechanisms: (1) High-precision operation: Our method operates at 90% precision with Recall@90P showing 79.3-90.1% coverage, minimizing false positives that cause premature stopping; (2) Ensemble robustness: Using top 4 of 8 trained classifiers provides robustness against individual classifier errors; (3) Adaptive thresholds: Lower thresholds increase recall at efficiency cost, allowing flexible conservativeness-efficiency trade-offs; (4) Fallback guarantee: If no chunk reaches sufficient confidence, we process the entire context, ensuring no information loss in edge cases.

Dependence on gold answer spans limiting applicability to open-ended tasks & Extension of synthetic labels to more complex scenarios

Thank you for the excellent question! Applying our method to scenarios without explicit answer locations is indeed a relevant research direction. We attempted to provide several solutions. In Appendix A.4, we conduct experiments on synthetic labels and find that GPT-4o-generated synthetic labels achieve competitive performance (F1: 84.6-87.0 vs 88.3-89.8 for original labels), enabling extension beyond factoid QA. For larger models (14B+), our self-prompting approach (L180-185 in §2.3) eliminates dependence on labeled data entirely, with Table 1 showing F1 scores of 78.3-83.1. The emergence of self-assessment capabilities also suggests the potential of our method to handle more complex scenarios with a training-free alternative.

For subjective tasks, we believe that the model's internal representations still encode when it has gathered sufficient information to form a coherent response, even if that information is distributed across the document or the answer is not explicitly located in the context. The key point is that sufficiency is ultimately a property of the model's understanding, not just the dataset structure. We left this question as a direction for further work but discussed some preliminary findings and ideas under an in-context-learning setting (Appendix G), which does not have explicit answers located in the context.

Automatic threshold selection for different models

This excellent suggestion addresses a key practical deployment challenge. Currently, threshold τ is selected on validation sets per model following standard hyperparameter tuning practices. However, the monotonic confidence progression (Figure 6) suggests potential for automatic threshold adaptation based on confidence curves, which is a valuable direction we acknowledge for future work. Thank you for highlighting this practical insight that could enhance deployment scalability of our method!

Thank you once again for your thoughtful engagement with our work. We appreciate your constructive feedback and look forward to incorporating these insights to further enhance our work.

Best regards,

The Authors

Thank you for your responses that are both clear and concise. It believe this work should be accepted therefore I keep my score.

Dear Reviewer xgYF,

Thank you immensely for the positive assessment of our work! We really appreciate the time and effort you invested in reviewing our work and providing valuable feedback!

Best regards,

Authors

Dear Reviewer 6R5V,

We sincerely thank you for your thorough and constructive review. We are grateful for your positive assessment of our paper's strengths, including the clear problem presentation, strong arguments, comprehensive experiments, and robust results. We especially appreciate your recognition that our work builds “a good end to end story on the effectiveness of this technique” and that you found our approach technically rigorous with no major weaknesses.

We would like to address your questions as follows:

Has there been any analysis done on whether tasks actually need early stopping? What about tasks where questions come at the end of the contextual information?

Thank you for this insightful question. We acknowledge that there are tasks that may not benefit from early stopping (eg: passage rewriting, summarization). However, this study focuses on tasks that would benefit from early stopping, which represents a substantial portion of real-world applications (eg: question answering, information retrieval, or fact verification).

A key advantage of our method is the potential to handle both scenarios naturally. Unlike other compression methods that compress regardless of the task, our approach can process the full context when necessary. When all information is crucial for a task, our sufficiency classifier would not trigger early stopping, effectively using the entire input.

Regarding tasks where questions appear at the end of the contextual information: our method is designed to detect when sufficient information has been processed regardless of where the key information appears. In principle, this can be enabled by mixing in training data where we know the full context is needed and training the classifier accordingly. Analyzing whether tasks need early stopping and detecting them explicitly would be a valuable future research direction that could further enhance the applicability of our approach - thank you for pointing out this interesting direction!

How is this more “dynamic” than other methods when you still fix parameters like chunking type and threshold value?

Thank you for this excellent question! We acknowledge this similarity and appreciate the opportunity to clarify our terminology. The key distinction lies in the compression behavior, not the parameter setting approach. In static methods like LLMLingua, a fixed compression ratio is applied uniformly, where a compression ratio of 50% results every input gets compressed to exactly half its original length regardless of content. In contrast, our method uses static hyperparameters to enable content-adaptive compression decisions. Some inputs may receive minimal compression (eg: 5%) while others may be heavily compressed (eg: 95%), with the actual compression amount determined by each input's specific information density and complexity.

The “dynamicity” we refer to is that each input receives different treatment based on its content, whereas other methods compress every input to the same predetermined level. The compression ratios we report are averages across all examples, not fixed targets applied uniformly. We acknowledge that “dynamic” may be misleading terminology and would be happy to use “content-adaptive” or “input-specific” to better describe this distinction.

Since larger models already know sufficiency signals, has there been study to let the model predict sufficiency using special tokens instead of separate classifiers?

Thank you for this excellent observation! The self-prompting (L136-142) method shows the scaling phenomenon as you mentioned. However, even for large models, the probing approach consistently outperforms self-prompting (91.1 vs. 83.1 F1 for 70B model in Table 1). This suggests that while models develop some intrinsic sufficiency awareness, their internal representations contain richer signals than what they can explicitly verbalize.

The special token approach you suggest is intriguing and represents a natural extension. Our probing results (Figure 2) demonstrate that sufficiency signals are most concentrated in specific attention heads of middle layers, which could guide the design of such special tokens. We view this as promising future work that could eliminate the lightweight classifier overhead entirely.

Is there analysis of the distribution of sufficiency chunk index across datasets? Any bias in the distribution of sufficiency signals?

Thank you for this important question! Our datasets are carefully balanced by design (Appendix A.1, Table 7-8), where the gold locations follow uniform distributions across all datasets (mean ≈ 0.50, with standard deviations of 0.25-0.28). This balanced design ensures that ~50% of chunks are classified as “insufficient” and ~50% as “sufficient”. The uniform distribution was intentionally created to provide fair evaluation across different information positioning scenarios.

Our confidence progression analysis (Figure 4) shows that the model's sufficiency confidence increases monotonically as more relevant information is processed, regardless of where that information appears in the context. This validates that our approach can handle diverse information distributions rather than simply learning positional biases.

We also tested our method on naturally occurring datasets where information placement wasn't controlled (see Appendix A.3 on synthetic labels), demonstrating robustness beyond balanced evaluation setup.

We appreciate your thoughtful engagement with our work. Your questions and feedback have allowed us to clarify key aspects of our approach and have highlighted important directions for future research. Thank you once again for your valuable feedback!

Best regards,

The Authors

Dear Reviewer r53E,

Thank you for your insightful reviews and comments! We appreciate your recognition of the novel paradigm and comprehensive evaluation of our work. We would like to address your concerns and questions as follows:

Dataset Bias and Data Distribution

You raise an important concern about potential bias for data distribution. Due to page limits, we had to move the dataset details to Appendix. However, we carefully designed our evaluation to avoid this bias. As detailed in Appendix A.1 (Table 6, 7), the gold location (answer position) follows a uniform distribution across all datasets:

• Mean gold location: 0.49-0.51 (middle of context)
• Standard deviation: 0.25-0.28 (wide spread)
• Range: 0.01-0.99 (covers entire context span)

This uniform distribution ensures approximately 50% of chunks are insufficient (before answer) and 50% are sufficient (after answer), creating a balanced evaluation that prevents the bias you mentioned. We also discuss dataset balance in Appendix A.2 and the collection process in Appendix A.3 in detail.

Regarding token counts: Our contexts range from 0.5K-40K tokens (mean ~23K for long datasets, ~2K for short), with detailed statistics also in Tables 6-7 of Appendix A.1.

Chunk Size Impact on Classification

Thank you for this important question! To clarify, our chunks are cumulative and maintained through KV cache reuse. When we process a new chunk, it contains all previous chunks plus the new segment, not just the isolated segment alone. This is a crucial distinction that ensures we preserve all contextual information throughout the processing.

Table 4 in Section 4.2 demonstrates that smaller chunks actually improve classification performance, with sentence-level chunking achieving F1=96.8 compared to 88.3 for 10% chunking. This is because smaller chunks allow for more frequent sufficiency checks, enabling earlier detection of relevant information. However, smaller chunks increase computational overhead due to more frequent forward passes. Therefore, we chose 10% chunking as the optimal balance between accuracy and efficiency.

Left-to-Right Processing

Thank you for this excellent question. While left-to-right processing may limit efficiency in some cases, our method offers a couple other advantages. Due to page limits, we had to move the discussion on the processing order to Appendix. However, as discussed in Appendix B.3, our approach offers several advantages compared to other processing orders: it maintains semantic coherence by preserving natural left-to-right text flow, enables KV cache reuse for computational efficiency, and most importantly, this processing order aligns with how current transformer models are trained and operate in practice.

While alternative processing orders could be explored, our current approach still achieves significant improvements with a 3.4% accuracy gain and 1.33X token reduction compared to static methods. We believe investigating optimal processing orders remains an interesting direction for future work, but our current results demonstrate the effectiveness of the left-to-right approach.

Information Loss Analysis

Thank you for this excellent suggestion! To comprehensively understand the tradeoffs between information loss and performance in our methodology, we would like to highlight the analysis in Sections 4.1 (Classification Threshold) and 4.2 (Chunking Strategy). However, the most compelling evidence that our method preserves rather than loses critical information is the consistent performance improvement. Specifically, we see a 3.4% average accuracy improvement (Tables 2-3), indicating our method helps models focus on relevant content rather than discarding important information.

To address concerns about false positives that could potentially lead to information loss, we report Recall at Precision 90-98% (Figures 4, 10, 11). Our threshold mechanism allows trading off precision for efficiency, where higher precision settings ensure no information loss at the cost of lower efficiency. Additionally, Figure 6 shows that models accumulate useful signals progressively, suggesting our approach captures relevant information effectively rather than making premature cutoff decisions.

Why not chunk input tokens with overlaps?

Thank you for this important question! We would like to clarify that each chunk extends the context incrementally (chunk₁, chunk₁+₂, chunk₁+₂+₃, etc.), which allows us to reuse the KV cache and avoid redundant computation of previously processed tokens. This cumulative approach maintains semantic continuity by preserving natural text flow without creating artificial boundaries that could disrupt understanding. On the other hand, overlapping chunks would require recomputing activations for the same tokens multiple times, which would negate our computational efficiency gains while disrupting the cumulative context semantics. The non-overlapping design ensures that each new chunk only requires computing activations for the truly new tokens, maximizing both computational efficiency and contextual coherence.

”Teaching” Terminology Confusion

This is an excellent point! Our method doesn't “teach” LLMs new capabilities but rather leverages existing internal representations. We will use “enabling” or “allowing” to avoid confusion.

We appreciate your suggestions and will incorporate them to enhance our work further. Thank you once again for your valuable feedback!

Best regards,

The Authors