Overcoming Long Context Limitations of State Space Models via Context Dependent Sparse Attention

We sincerely thank the reviewer for the thoughtful and positive assessment of our work. We greatly appreciate your recognition of the significance of our contributions, including the relevance of our study to subquadratic sequence modeling, the value of the joint recall task as a synthetic benchmark, and the motivation behind our proposed hybrid sparse attention mechanism. Your feedback encourages our continued exploration in this important direction. Below, we address your specific questions and concerns in detail.

W1: On the motivation for proposing joint recall

Thank you for raising this important point about the motivation of our proposed joint recall task.

The core motivation for joint recall lies in testing context-dependent memory, a key aspect of real-world language understanding that is largely absent in prior synthetic tasks such as associative recall. While associative recall benchmarks evaluate whether a model can retrieve a value when queried with a corresponding key, they assume a context-independent one-to-one key-value mapping. This design does not reflect the nature of natural language, where context disambiguates meaning.

To illustrate this, we provide the following example:

• Associative Recall:

  • Context: “The oceans are blue, and snow is white.”
  • Query: “What color is snow?”
  • Answer: “White.”

• Joint Recall:

  • Context:
    • “In the U.S., people pay with dollars and drive on the right side of the road.”
    • “In the U.K., people pay with pounds and drive on the left side of the road.”
  • Query: “Which side of the road do people drive on in the U.S.?”
  • Answer: “Right.”

From this example, we can see that associative recall does not account for context. Joint recall extends associative recall by incorporating context-dependency into key-value associations. For example, while associative recall may map "pay with" to either "dollar" or "pound", joint recall allows it to map to "dollar" in the U.S. and "pound" in the U.K., depending on context. This makes joint recall a more realistic and rigorous synthetic task for both theoretical analysis and empirical benchmark for long-context modeling.

Regarding the suggestion of using multiple keys with different values per key, we agree that such configurations have been explored in benchmarks like Ruler [14]. For reference, we have already tested the Ruler benchmark in our main experiment, including those tasks as well. However, those designs do not evaluate context-sensitive resolution, where the same query token (e.g., “pay with”) may map to different values depending on contextual modifiers (e.g., country). This is precisely the ability we aim to isolate and test with joint recall.

Thus, our task uniquely highlights the limitation of SSMs in capturing input-conditioned compositional associations, and it provides a controlled yet challenging environment for studying improvements from hybrid architectures like LSH+KC.

W2: On the motivation of introducing LSH

Thank you for this insightful question. We agree that compressing or merging keys can be beneficial as indicated in related works (e.g., SE-Attention, Native Sparse Attention). Below, we clarify our motivation for introducing LSH attention and discuss its relationship to key compression.

1. Theoretical necessity of input-dependent sparsity for hybrid sparse attention.

We introduce the following new proposition 4: There does not exist a 2-layer auto-regressive hybrid model consisting of an SSM layer followed by an input-independent sparse attention layer, which can solve the joint recall task with $o(L^2)$ time complexity, since it requires at least $O(L/k)$ SSM state dimensions, $k$ is the maximum number of keys that each query allowed to attend to in the sparse attention module, as defined in Eq.4.

The proof is provided at the end of the response to this point. This establishes that any sub-quadratic hybrid sparse attention must employ input-dependent sparse attention, i.e. sparse attention with a pattern that is conditioned on the query and key representations at inference time.

2. LSH as an instance of input-dependent sparse attention which solves joint recall.

We do not claim LSH attention to be the unique or universally optimal input-dependent mechanism. Rather, we show in Proposition 3 that LSH as a concrete instantiation of input-dependent sparse attention, provably suffices to solve joint recall when integrated with SSMs. By choosing LSH, we verify our theoretical result with a concrete, well-studied algorithm. Any other input-dependent scheme with similar properties would also satisfy the theory, and exploring those is an exciting avenue for future work.

3. LSH’s orthogonality to key compression.

It is true that compressing keys (via aggregation or learned pooling) can preserve downstream performance in many real-world tasks. In our synthetic joint recall construction, however, no redundancy is present by our construction (please refer to our proof of Proposition 3 in Appendix C.2). For clarity and analytical tractability, we therefore omit key compression from both our proofs and our empirical study. That said, we do not rule out combining LSH+KC with key-compression techniques on real-world data; such hybrid approaches might yield additional efficiency gains.

We hope this clarifies our choice of standard LSH attention: it serves as a concrete and well-established example of input-dependent sparse attention, and when integrated with SSMs, it provides a provable solution to the joint recall task.

proof of proposition 4. Consider a key given in the inquiry component of the auto-regressive joint recall task. The model is required to output the associated value token when this key token is provided as input. Taking the query representation from this key token, the sparse attention can attend to at most $k$ key representations from previous tokens, where the key representations are calculated based on the SSM state representations of the first layer. To solve the joint recall task, these $k$ key representations being attended must collectively encode the full context. Since the full context length is $O(L)$, by Corollary 1, $k$ state representations of the generalized SSM in the first layer must use at least $O(L/k)$ dimensions to collectively store the full context. Thus, the per-key computational cost required by the second-layer sparse attention is $O(k \cdot (L/k)) = O(L)$, and therefore the total time complexity is $O(L^2)$.

W3 / Q3 / Q4: On the related work: B’MOJO, SE-Attn, and NSA

We would like to clarify that our motivation differs in key ways from the works mentioned. NSA targets hardware efficiency via compression-based sparse attention. SE-Attn improves retrieval in hybrid SSMs through relevance-based memory access. B’MOJO proposes a realization-theoretic architecture for transductive inference. In contrast, our method is motivated by an expressiveness analysis grounded in the joint recall task, a synthetic benchmark designed to capture key challenges in context-dependency modeling. This analysis reveals a fundamental limitation of SSMs under sub-quadratic constraints. To address it, we introduce a method which combines SSM with input-dependent sparse attention, instantiated by LSH attention, and we present both the theoretical analysis and empirical verification on the advantage of this method.

We also note that none of NSA, SE-Attention, or B’MOJO have released their code. As a result, a direct empirical comparison is currently not feasible. However, we will include a more detailed discussion of these works in the related work section.

W4: On the hardware-aware implementation

While we agree that hardware-aware implementation is crucial for scalability and plan to explore it in future work, our current focus is on establishing the theoretical necessity and empirical effectiveness of input-dependent sparse attention. We believe this contribution is significant in its own right, as it identifies a fundamental limitation of SSMs and provides a provably expressive and empirically validated solution.

Q1: On the hashing implementation

Our choice to implement both the argmax and sign-bit binning rules is by no means arbitrary, but follows directly from the two most widely adopted LSH-attention schemes in the literature: argmax binning (represented by Reformer [16], Learning-to-hash attention [C.1]), and sign-bit binning (represented by MemoryFormer [C.2], HashAttention [C.3]).

Q2: On the random hash projection matrix

Following Reformer [16], we use a random matrix for hash projection. In our implementation, we randomly sample a matrix at each step during training, and fix a random matrix during inference. We do not perform any tuning or adaptation of the random matrix at inference time.

Q3: On Samba performance

Thank you for the question. While we have not fully understood the root cause of this discrepancy, we ensured that the comparison was fair by using consistent training settings under the FLA implementation [C.4]. The details of our setup and efforts to ensure fairness are provided in Appendix D.1. We will continue investigating this behavior to better understand the performance gap in future work.

Minor: broken reference

We’ll correct it in our future versions.

We once again thank you for your thoughtful and constructive feedback. Your comments have truly helped us clarify our motivation, contributions, and distinctions from prior work, and we appreciate the opportunity to improve our future research as well as the manuscript through your insights.

[C.1] Sparse Attention with Learning to Hash. ICLR 2022.

[C.2] MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers. NeurIPS 2024.

[C.3] HashAttention: Semantic Sparsity for Faster Inference. ICML 2025.

[C.4] fla-org/flash-linear-attention. github.

b. Conceptual comparison with compression-based sparse attention

Typically, compression-based sparse attention rely on two key assumptions:

1. The input context is compressible, and
2. The context exhibits predefined structural properties (e.g., block boundaries for block compression), which is suitable for compression.

In contrast, we do not assume that the context is compressible or that it follows a known structure. Instead, we focus on improving a model’s general ability to compose multiple, spatially distributed cues in the context. This is theoretically grounded by the joint recall task, where we show that pure SSMs face fundamental limitations, and where our hybrid LSH+KC mechanism overcomes those limitations by introducing input-dependent sparse attention. Due to these differences in assumptions and goals, we do not view compression-based models as directly comparable baselines in our study.

A minor point on technique: we note that many prior works (including NSA, SE-Attn, Samba, and B’MOJO) incorporate sliding window attention as a core component. However, in our study, both theoretical analysis and empirical results suggest that sliding window attention might not be the best solution for hybrid sparse attention.

Last but not least, we agree that conceptual comparison is important. We will revise the manuscript to include a more thorough discussion of these works in the related work section.

c. Effective gains on new results

d. Response to Reviewer YZT8: Scope of empirical support

We thank the reviewer for raising these important concerns regarding the scale and impact of our empirical results.

First, we acknowledge that the performance gains reported on the 790M model are modest. However, we would like to clarify that the 790M model under continual pre-training is still far from convergence. Due to the formatting constraints in this rebuttal phase, we are unable to include the training loss curves, but we observed a consistent downward trend that suggests further improvements with continued training.

Second, as in the original Mamba paper, we followed the recommended practice of using a smaller learning rate for continual pre-training at this scale. While this promotes stable learning, it also increases the number of training steps required for meaningful gains, posing a significant computational burden under academic resource constraints.

Third, it's important to highlight that LongBench is a challenging benchmark where even well-designed architectural changes often yield limited gains. As we noted in our response to Reviewer YZT8’s comment W1, even a 130M hybrid model augmented with full attention achieves only marginal improvements over the Mamba baseline. This underscores the difficulty of improving LongBench performance, especially in continual pre-training settings.

Fourth, and more broadly, we would like to clarify that the primary goal of our empirical study is to validate the our theoretical analysis. Across all our settings, spanning both 130M and 790M models, The Pile and TxT-360 continual pre-training datasets, and evaluations on RULER, LongBench, as well as the joint recall task, we consistently observe the following results:

1. Adding input-independent sparse attention (e.g., sliding window) to SSMs results in negligible gains.
2. Adding only LSH or only KC attention is similarly ineffective.
3. Only the combination of LSH and KC attention yields meaningful improvements, in line with our expressiveness analysis.

Lastly, we acknowledge the reviewer’s concern about limited architecture diversity. Ideally, we would explore from-scratch pre-training as the most principled route to test generalization, but this is beyond our current academic resources. As a result, we rely on continual pre-training, which is inherently constrained by the properties of the base checkpoint (e.g., Mamba’s 2K pre-training context length). Despite this, we believe our findings offer general valuable insight into the design of sub-quadratic hybrid architectures and help guide future exploration at scale.

Once again, we sincerely thank you for your thoughtful and detailed feedback. Your comments have helped us clarify our contributions, better position our work within the literature, and identify valuable directions for further improvement. We will incorporate the suggested revisions in the final version of the manuscript.

We sincerely thank you for the thoughtful and constructive comment. We'll definitely revise our paper for better clarification of joint recall. Below, we address your new concerns in detail.

a. Empirical validation for the effectiveness of joint recall

We appreciate the reviewer’s thoughtful concern regarding the diagnostic value of the joint recall benchmark in comparison to existing long-context evaluations such as RULER. Our key motivation for introducing joint recall is to expose a fundamentally different failure mode in long-context models, specifically, the inability to integrate multiple, spatially distributed contextual cues to recover a value, rather than simply locating a single relevant key token as in associative recall or RULER.

From the RULER paper, we see that RULER benchmark is already saturated, with multiple models achieving an average accuracy > 90%. However, our natural language version of joint recall remains challenging for the most powerful LLM today. Owing to the time constraint of the author-reviewer discussion period, we can only report a case study based on ChatGPT-o3.

In this case study, we consider 5-keys joint recall. The 5 key sets are:

1. Years (2020 - 2024);
2. Months (January - August);
3. Days of the week (Monday - Sunday);
4. Time of the day (morning, afternoon and evening);
5. Student names (8 common names).

And the value set is 10 common high school subjects. We consider 2 settings to organize the context. The first setting is:

<input>
Keep in memory:
In 2020:
  In January:
    On Mondays:
      In the morning:
        Susan studied Geography;
        John studied History;
        ...
      In the afternoon:
        ...
      In the evening:
        ...
    On Tuesdays:
      ...
  In February:
    ...
In 2021:
  ...

Now let's recall:
In June 2023, On Wednesday mornings, Thomas studied:

<answer>
Mathematics.

The second setting is:

<input>
Keep in memory:
In January 2020:
  On Mondays:
    In the morning:
      Susan studied Geography;
      John studied History;
      ...
    In the afternoon:
      ...
    In the evening:
      ...
  On Tuesdays:
    ...
In February 2020:
  ...

Now let's recall:
In June 2023, On Wednesday mornings, Thomas studied:

<answer>
Mathematics.

The only difference between the two settings is that the second collapses the year and month levels (e.g., “January 2020”), while the first retains full hierarchical structure. The key-value associations are randomly sampled once and kept identical across the two settings to ensure comparability. We generate 10 data points for both settings and evaluate ChatGPT-o3’s one-shot response manually via the ChatGPT temporary chat web interface. The data length is about 32K tokens according to the OpenAI online tokenizer.

Surprisingly, we observe that ChatGPT-o3 correctly answers only 5 out of 10 examples in the first setting, but successfully answers all 10 in the second setting. Interestingly, the input length of the second setting is even slightly longer than the first setting, since the year is repeated after each month.

From this case study, we draw two preliminary conclusions: (1) even today’s most powerful LLMs are not yet capable of perfectly solving the natural language version of joint recall task; and (2) their performance is limited not just by context length, but also by their ability to integrate and compose multiple pieces of context effectively.

Therefore, joint recall is not just another synthetic variant of tasks in RULER. RULER mainly focuses on context length, we are among the first to show that context structure also matters. Note that this point is not just about the limitations of SSMs. It can also provide insight for the broader field of long-context modeling.

We sincerely thank the reviewer for encouraging us to further investigate this direction, and we will report more formal and comprehensive experiments in the final version of our paper.

We sincerely thank you for the thoughtful and encouraging feedback. We are glad that you found the design of our LSH+KC attention mechanism to be novel and well-motivated.

Below, we address your specific questions and concerns in detail.

W3: The scale of the experiments are small.

We have newly conducted continual pre-training based on the 790M Mamba official checkpoint. These larger-scale experiments confirm the effectiveness of our proposed LSH+KC method on real-world long-context benchmarks. For these experiments, we use the TxT-360 corpus (not the same as The Pile used in our paper) as the continual pre-training dataset. We continual pre-train for 50,000 steps with a context length of 2K, which is followed with instruction tuning using UltraChat (the same as in the paper) for 10,000 steps with a context length of 2K. We also train new 130M models in this setting. The new results on Ruler and LongBench are presented below:

Table R.1: 4K Ruler results based on 790M Mamba

Table R.2: LongBench results based on 790M Mamba

Table R.3: LongBench results based on new 130M Mamba

W1: The paper seems rushly written.

We sincerely apologize for that. After submission, we have substantially revised the manuscript to improve clarity, organization, and technical precision.

W1: Experiment results are not complete; Q1: Add the QA2 task in Ruler benchmark.

Thank you for pointing this out. The results are provided at Table R.1. These 790M models were pre-trained with a 2K context length, so the 4K evaluation tests their extrapolation ability. Notably, our LSH+KC architecture still outperforms all baselines on average Ruler performance (including QA2), highlighting its robustness to longer contexts.

W1: Experiment results are not complete; Q2: Add the hybrid dense attention baseline. Q6: Add the SW+LSH baseline. Q8: Include the standard deviation on joint recall.

Across all benchmarks, our proposed Mamba+LSH+KC consistently outperforms Mamba+LSH+SW and narrows the gap between SSM and hybrid full attention. For the joint recall experiments, due to the rebuttal character limitations, please refer to our rebuttal of W2/Q2 to the reviewer MdxV. The Ruler results are provided in the following table:

And the LongBench results are provided in the following table:

W1: The experiment setup and evaluation settings are missing.

We apologize if the presentation was unclear and would like to clarify that:

• We describe the baseline setup and evaluation procedure in Section 5, with additional training and implementation details provided in Appendix D.
• For the joint recall task, evaluation settings are detailed in Section 5.1.
• For Ruler [14] and LongBench [3], we follow their standard evaluation settings.

W1: Technical details on KC attention are not clear.

We provide the technical description of KC attention in Section 4.2, including the method, mathematical formulation, and training objective. The MLP is the simplest 3-layer MLP with ReLU activation.

W1: The results on LongBench seem not significant.

We emphasize that LongBench poses significant challenges for long-context modeling and demands substantial training to perform well. Our largest continual pretraining run, a 790M model trained on approximately 8B tokens, remains far from convergence. Nevertheless, our proposed LSH+KC architecture consistently outperforms all baselines on LongBench across three evaluation settings (Table 3, R.2, R.3), underscoring its practical effectiveness even under limited academic training resources.

W2: It is unclear how effective KC it is in real-world scenario.

We would like to clarify that KC attention is not intended to serve as a stand-alone module, but rather as a complementary component to enhance the expressiveness of LSH attention. As shown in our experiments (Tables 2, 3, R.1, R.2, R.3), our hybrid LSH+KC attention consistently outperforms both the base SSMs and all other SSM variants augmented with input-independent sparse attention, including sliding window attention, A-shaped attention (sliding window + sink) and sliding window + dilated attention. This advantage holds across both synthetic and real-world long-context benchmarks, validating the effectiveness of KC as part of a larger input-dependent attention design.

W4: Missing Citations.

The two referenced papers contribute to sparse attention from an efficiency perspective. The first presents a Triton-based kernel that accelerates both QK-sparse and hash-sparse attention. The second introduces Sparse Modular Activation, which dynamically gates attention modules per token in hybrid SSM-attention models.

In contrast, our work is motivated by the goal of improving long-context modeling through a deeper understanding of its expressiveness constraints. First, building on a theoretical analysis of the joint-recall task, we identify a fundamental limitation in the representational capacity of SSMs and motivate the integration of LSH attention to address this challenge. Second, we further propose KC attention to address the expressiveness limitation of LSH. Finally, we integrate our method into modern state-space models, specifically Mamba and Mamba-2, demonstrating its empirical benefits at scale.

We appreciate the reviewer’s pointers to these works and will add a subsection in Related Work to discuss related sparse attentions.

Q3: Which dataset did you use for continual pretraining?

We followed the original Mamba setup and used The Pile for continual pretraining, as noted in Appendix D.2.

Q4: How sensitive are the continual training results on the scalers of the auxiliary loss in KC attention?

Thank you for bringing this up. While we did not extensively tune this scaling factor, we observed that choosing a suitable value is important for achieving good downstream performance. Here are the LongBench results based on different scaler values:

Q5: How many parameters does the MLP have for key scoring?

The MLP used for key scoring in KC attention is lightweight relative to the base model. We provide the comparison of the number of parameters in the following table:

This demonstrates that LSH+KC attention introduces minimal overhead while delivering meaningful performance improvements.

Q7: How many test samples are in the joint recall synthetic dataset?

As noted in Line 232, the joint recall synthetic dataset contains 10,000 test samples.

We sincerely thank the reviewer once again for the constructive feedback and insightful questions, which have helped us strengthen the clarity, depth, and presentation of our work.

We sincerely thank the reviewer for the thoughtful and detailed feedback, as well as for increasing the score. Your suggestions, particularly on running more comprehensive experiments and improving writing clarity, are deeply appreciated. We are committed to revising the paper based on your advice and are grateful for your engagement and support.

We sincerely thank you for your thoughtful evaluation of our work. We greatly appreciate the positive feedback regarding the novelty of our hybrid sparse attention, the clarity of our theoretical analysis, and the solidity of the empirical results. We are also glad that the introduction of the joint recall task was seen as a meaningful contribution to theoretical analysis and empirical benchmark for long-context modeling. Below, we address your specific questions and concerns in detail.

W1/Q1: On the potential societal impacts of our work

Thank you for raising this important point regarding broader societal impacts.

Our work focuses on fundamental architectural improvements for long-context sequence modeling, particularly by enhancing the theoretical and empirical understanding of hybrid sparse attention in state-space models. The goal is to advance the efficiency and expressiveness of machine learning models at the architectural level, a domain that is general-purpose and infrastructure-oriented rather than application-specific.

As such, our contributions do not involve direct interaction with human users, personal data, or deployment in potentially risky domains (e.g., medical, legal, or surveillance applications). Consequently, we believe our work does not introduce significant ethical risks or societal concerns in its current scope.

Meanwhile, we agree that architectural advances may eventually be used in downstream applications. Therefore, we fully support the community’s efforts to consider responsible deployment and fairness at later stages of the ML pipeline. We have updated the Conclusion section to explicitly clarify the scope and neutrality of our contributions with respect to societal impacts.

W2/Q2: Statistical significance report

We appreciate your suggestion regarding the inclusion of statistical significance measures.

For the Ruler [14] and LongBench [3] benchmarks, we follow the established evaluation protocol used in prior work, which typically reports single-run results without error bars. We adopted the same reporting standard to ensure comparability and consistency.

For our proposed joint recall task, we conducted experiments using three random training seeds and summarize the average results with standard deviations in the table below:

Two new baselines are added to this table: hybrid LSH + sliding window (SW) attention and hybrid full attention. As observed, the variance in performance can be relatively high. This behavior is inherent to the nature of the synthetic task: the model either learns the underlying rule and achieves near-perfect accuracy, or fails to do so and performs poorly when being stuck at a local minimum. This leads to a bimodal outcome distribution, which inflates the standard deviation.

W3: Data availability and code reproducibility

We have provided the link to the full codebase at multiple locations in the paper (Lines 248, 479, and 518), and we will further highlight these links in the final version for better visibility.

Both the joint recall synthetic task and the continual pre-training experiments are fully reproducible by running the scripts in our repository. For the continual pre-training experiments in the paper, we use only publicly available datasets:

• Continual pre-training follows Mamba [10] using The Pile dataset.
• Instruction tuning follows Mamba-Chat [20] using UltraChat.
• Evaluation is performed on Ruler [14] and LongBench [3], both of which are publicly accessible.

We provide a detailed description of training setups in Appendix D. We hope this clarifies the reproducibility of our experiments.

Once again, we sincerely thank you for the helpful comments and suggestions. We sincerely hope that our rebuttal resolves the concern you highlighted, and we are grateful for your time and insights.

Dear Reviewer MdxV,

We appreciate the time and effort you put into reviewing our submission. We would be grateful if you could take a moment to consider our rebuttals and share any additional thoughts you might have. Your insights would be extremely valuable to us.

Best, Anonymous authors

Thanks for the detailed response and the efforts the authors provided. The rebuttal addressed most concerns satisfactorily. Here are some further suggestions.

• More discussion with related work. The authors could further discuss or add these baselines, like SE-Attn [1], B’MOJO [2], and NSA [3]. Given their direct relevance (hybrid span expansion and hardware-aligned sparsity), additional comparisons would make the manuscript more convincing with claims of designed methods, e.g., the hardware-aligned sparse methods in NSA are similar to the proposed method.

• LSH design choices. Do the authors further explore or clarify whether the random matrix HH is fixed vs. resampled, the number of hash rounds/bins, or bucket balancing—key for stability and reproducibility in LSH attention.

• More Comparison on popular real-world benchmarks. In the manuscript and rebuttal responses, the authors have provided benchmark results on Ruler and LongBench, which are convincing to verify the efficiency. The authors could further provide commonly used benchmarks like Wikitext, common-sense reasoning tasks, and question-answering tasks, as conducted in previous correlated works like [4].

Reference

[1] Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models. NeuS, 2025.

[2] B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory. NeurIPS, 2024.

[3] Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. ACL, 2025.

[4] Gated Linear Attention Transformers with Hardware-Efficient Training. ICML, 2024.

We thank the reviewer for the constructive feedback and helpful suggestions:

• Related Work and Baselines. We thank the reviewer for pointing out SE-Attn [1], B’MOJO [2], and NSA [3]. We agree that these works are relevant and will revise the manuscript to include a more thorough discussion of them in the related work section. We will also provide empirical comparisons once their official implementations become publicly available.
• LSH Design Choices. We appreciate the suggestion to clarify our LSH configuration. In our implementation, the random projection matrix $H$ is resampled at each training step to improve robustness, and fixed during inference to enable consistent auto-regressive cache representation. We use a single hash round with $k$ hash bins, where $k$ is the sparsity constraint (the maximal number of keys a query can attend to) to maintain efficiency. While identifying the optimal LSH setup is not the primary focus of this work, we will explore these design variants (e.g., more rounds or bucket balancing) when computational resources permit.
• Additional Benchmarks. If the reviewer is referring to the benchmark in Table 2 of GLA [4], we note that we have already included the same evaluation in Appendix E. We did not place it in the main text because these benchmarks have relatively short context lengths, whereas our focus is on long-context modeling. Nevertheless, we will make this connection explicit in the revision so that readers can easily locate the results.

We again sincerely thank the reviewer for the thoughtful and detailed feedback. We are committed to revising the paper based on your advice and are grateful for your engagement and support.