LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field Enlargement

Tab. A: Benchmarking results on LongBench dataset with 1.4B Mamba models. This table compares the LongBench performance achieved by the vanilla Mamba model, DeciMamba, as well as the two proposed methods that specially handle the global channels stemming from our insight. To enhance readability, the table is divided into two sections (top and bottom), with all values expressed as percentages.

Tab.B: Perplexity on the PG19 dataset with 1.4B Mamba models. This table compares the language modeling performance of DeciMamba as well as the two proposed decay adjustment methods.

As observed in Tab. A and Tab. B, the alternative method (“Ours: Normalization”) can consistently improve Mamba’s performance for long-context tasks. For instance, it brings a 5.4% higher average accuracy on LongBench (as shown in the last column of the bottom half of Tab. A) and consistently reduced perplexity compared to DeciMamba on the PG19 dataset (as shown in the second and last row of Tab. B).

This set of experiments further validates the effectiveness and generality of our insight. We believe this insight could shed light on future research on other decay adjustment methods and innovations to enhance SSMs’ long-context capabilities.

Q2: LongMamba shows some improvements for long contexts, but it still seems to be somewhat behind Transformer-based LLMs designed for long contexts. If training were allowed, would the performance be further enhanced?

Thank you for this great question! We have applied our LongMamba to a 7B Mamba model (i.e., Falcon-Mamba 7B [a]) and compared it with other 7B Transformer [b,c,d] models on LongBench. Specifically, the LongBench results of Transformer models [b,c,d] are the reported ones in the LongBench paper.

We greatly appreciate your positive feedback and constructive suggestions/questions, and have addressed each of your concerns and suggestions below:

Q1: I find that LongMamba's approach—pruning less important tokens to increase the global channel's receptive field—seems somewhat incremental compared to DeciMamba.

We humbly clarify that the key contribution of LongMamba lies in the insight distinguishing global and local channels in Mamba, which serves as the foundation for enhancing its long-context capabilities. While we use token filtering to implement this insight, we emphasize that it is just one of many possible approaches to leverage or implement our insight, showcasing the broader applicability and originality of our contribution. Specifically, our new contributions beyond DeciMamba are as follows:

1. Key Insight:

We discover that the exponential hidden state decay of the global channels is the primary cause of Mamba’s constrained receptive field and limited performance on long-context tasks. This finding is also highly recognized by other reviewers: Reviewer P417 remarked, "The finding about the categroies of channels of Mamba models (local and global) is interesting and meaningful, which could bring insight to the community for better understanding the Mamba model's behavior", and Reviewer GwXu stated, “the main result is very interesting.”

2. Token Filtering Implementation:

Leveraging this key insight, we propose specially handling the global channels to enlarge their receptive field by applying selective token filtering exclusively to the global channels while preserving the locality of the local channels. This strategy has outperformed DeciMamba and maintained the same language modeling perplexity on contexts 10x longer, further validating the significance and practicality of our key insight. This has also been recognized by other reviews, such as “Overall a simple mechanism that seemingly enables Mamba to generalize to much longer contexts” by Reviewer GwXu.

3. Alternative Implementation:

To further demonstrate the generality of our insight, during the rebuttal period, we implemented and validated an alternative method.

This method dubbed “decay normalization” adjusts the decay by normalizing $\Delta_t$ for all tokens in the same global channel with a scalar constant c (i.e., $\Delta’_t = \Delta_t / c$), and subsequently derives the adjusted $\bar{A}’_t$ and $\bar{B}’_t$ using Eq. 7 of the submitted manuscript. The factor $c=c(S)$ is computed to satisfy Eq. 16 in Sec. 5.2 of the submitted manuscript, similar to how we obtain $g(S)$.

We evaluated this alternative decay adjustment method on both the LongBench and PG19 tasks. The results are summarized in Tab. A and Tab. B below, where “Ours: Token Filtering” corresponds to the decay adjustment method described in the submitted manuscript, and “Ours: Normalization” represents the alternative approach proposed above.

Q3: It would be more convincing if the authors could provide an evaluation of the Ruler benchmark.

Thank you for the advice! Following your suggestion, we have added the experiments on the RULER benchmark. Specifically, we adopt the Mamba-1.4B model, which was pre-trained on 2k sequence length, and test its performance on both the 8k and 16k sequence lengths provided by the Ruler benchmark. The results are reported in Tab. D and E below, respectively.

We can observe that LongMamba demonstrates notable performance improvements across all kinds of long-context tasks from the Ruler benchmark, highlighting the effectiveness of the proposed framework in addressing diverse long-context understanding challenges.

Tab. D: Benchmark results on the RULER dataset under a 8192 context length using the 1.4B Mamba model.

Tab. E: Benchmark results on the RULER dataset under a 16384 context length using the 1.4B Mamba model.

For Tab. D and E, we generate 100 sequences for each task. Note that we omit the “niah_multikey_2” and “niah_multikey_3” tasks, because the Mamba-1.4B model achieves 0 accuracy on both tasks even with 2k context length

---

[a] Zuo, J., Velikanov, M., Rhaiem, D. E., Chahed, I., Belkada, Y., Kunsch, G., & Hacid, H. (2024). Falcon mamba: The first competitive attention-free 7b language model. arXiv preprint arXiv:2410.05355.

[b] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

[c] Nijkamp, E., Xie, T., Hayashi, H., Pang, B., Xia, C., Xing, C., Vig, J., Yavuz, S., Laban, P., Krause, B., Purushwalkam, S., Niu, T., Kryscinski, W., Murakhovs’ka, L., Choubey, P. K., Fabbri, A., Liu, Y., Meng, R., Tu, L., Bhat, M., Wu, C.-S., Savarese, S., Zhou, Y., Joty, S. R., & Xiong, C. (2023). Long sequence modeling with XGen: A 7B LLM trained on 8K input sequence length. arXiv preprint. arXiv:2309.03450

[d] Zheng, L., Chiang, W. L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., ... & Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 46595-46623.

[e] Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., & Jia, J. (2024). LongLoRA: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307

[f] Zhang, H. (2024). SinkLoRA: Enhanced efficiency and chat capabilities for long-context large language models. arXiv preprint arXiv:2406.05678

[g] Cai, R., Tian, Y., Wang, Z., & Chen, B. (2024). LoCoCo: Dropping in convolutions for long context compression. arXiv preprint arXiv:2406.05317

Tab. C: Benchmark 7B Mamba and Transformer models on LongBench. The table compares the performance of the vanilla 7B Falcon-Mamba model [a], LongMamba-enhanced Falcon-Mamba, Llama2-7B-chat-4k [b], XGen-7B-8k[c] and Vicuna-v1.5-7B-16k [d] on the long-context understanding tasks from LongBench. To enhance readability, the table is divided into two sections (top and bottom), with all values expressed as percentages.

We can draw the following observations:

1. The vanilla Mamba exhibits lower performance compared to Transformer models on long context understanding tasks. Specifically, Llama2-7B-chat-4k [b], XGen-7B-8k[c], and Vicuna-v1.5-7B-16k [d] achieve 2.4%, 1.3% and 6.7% higher average accuracy, respectively, compared to the vanilla 7B Falcon-Mamba model (refer to the last column in the bottom half of the table).
2. Our LongMamba can enhance the performance of vanilla Mamba models and narrows the gap with or even surpass Transformer models. As shown in the last column of the bottom half of the table, LongMamba achieves a 2.8% increase in average accuracy compared to the vanilla Mamba model, outperforming Llama2-7B-chat-4k [b] (+0.4%) and reducing the gap with Vicuna-v1.5-7B-16k [d] from 6.7% to 4.0%.

We believe that this is a positive signal for the research community, implying that Mamba's long-context capability can match, if not surpass, that of Transformers. Our LongMamba approach demonstrates the potential to unlock Mamba's long-context capabilities in a training-free manner, serving as a promising step toward activating these latent abilities and inspiring the development of more sophisticated long-context solutions for Mamba.

In addition, as you suggested, further long-context fine-tuning, which has been shown to be effective for Transformer models [e, f, g], has the potential to further narrow the performance gap. LongMamba is currently designed as a training-free technique for ease of use and can be integrated into the fine-tuning process or applied in a post-fine-tuning manner. Although we don’t have sufficient time to complete this exploration within the rebuttal period, this is a promising direction for future work. We will actively work on this and share our findings with the community once ready.

Q1: How does the LongMamba perform on the code category of LongBench?

Thank you for the question! We have provided this comparison with both the vanilla Mamba and DeciMamba on the LCC and RepoBench-P tasks (i.e., two coding tasks) in LongBench in Tab. B below. We can observe that LongMamba still achieves higher accuracy than both the vanilla Mamba and DeciMamba, consistently demonstrating its effectiveness. We will add the full benchmark on LongBench in the final version.

Tab. B: Benchmarking results on the coding tasks from LongBench with the 1.4B Mamba model. All values are expressed as percentages.

---

Q2: Does the token filtering process bring extra compute or memory cost?

Thank you for asking this important question! We clarify that the token filtering process in our method introduces negligible latency and memory overhead. Specifically, following your question, we have tested the latency and memory overhead of the token filtering process in our method using both 130M and 1.4B Mamba models, with a batch size of 1 and context lengths ranging from 2K to 128K tokens. We consistently observe a negligible (<5%) increase in both the prefilling and generation latency, as well as the peak memory usage (reported by the PyTorch memory profiler). This minimal impact is due to the highly efficient implementation of the token filtering operation, which selectively sets specific elements in the Δ tensor to zero. Since the Δ tensor is 16 times smaller than the SSM state, operations on it result in minimal additional computational or memory costs compared to the inference process.

We will clarify this in the final version.

---

Q3: How do the "local and global channels" behave on a larger Mamba model?

Thank you for the question! To answer your question, we have applied our method to the 7B Falcon-Mamba model [a], which is one of the best-performed Mamba models, adopting the same 1e-30 threshold for identifying global channels. The results are provided in Tab. A (benchmark on LongBench, in our response to W1 above) and Tab. C (benchmark on PG19 perplexity) below.

Tab. C: Perplexity on PG19 with different context lengths with 7B Falcon-Mamba [a].

The results in Tab. A and Tab. C consistently demonstrate that at the 7B model scale, LongMamba can still notably enhance the performance of Mamba models on long-context tasks. For example, LongMamba achieves a 2.8% increase in average accuracy compared to the vanilla 7B Mamba model on LongBench in Tab. A and enables consistently lower perplexity on PG19, especially under longer sequences, in Tab. C.

---

[a] Zuo, J., Velikanov, M., Rhaiem, D. E., Chahed, I., Belkada, Y., Kunsch, G., & Hacid, H. (2024). Falcon mamba: The first competitive attention-free 7b language model. arXiv preprint arXiv:2410.05355.

[b] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

[c] Nijkamp, E., Xie, T., Hayashi, H., Pang, B., Xia, C., Xing, C., Vig, J., Yavuz, S., Laban, P., Krause, B., Purushwalkam, S., Niu, T., Kryscinski, W., Murakhovs’ka, L., Choubey, P. K., Fabbri, A., Liu, Y., Meng, R., Tu, L., Bhat, M., Wu, C.-S., Savarese, S., Zhou, Y., Joty, S. R., & Xiong, C. (2023). Long sequence modeling with XGen: A 7B LLM trained on 8K input sequence length. arXiv preprint. arXiv:2309.03450

[d] Zheng, L., Chiang, W. L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., ... & Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 46595-46623.

We greatly appreciate your positive feedback and constructive suggestions/questions for our work, and have addressed each of your comments in detail below:

W1: I believe the experiments section can be improved by adding direct comparison with other long-context models beyond Mamba variants.

Thank you for this great suggestion! We have applied our LongMamba to a 7B Mamba model (i.e., Falcon-Mamba 7B [a]) and compared it with other 7B Transformer [b,c,d] models on LongBench. Specifically, the LongBench results of Transformer models [b,c,d] are the reported ones in the LongBench paper.

Tab. A: Benchmark 7B Mamba and Transformer models on LongBench: The table compares the performance of the vanilla 7B Falcon-Mamba model [a], LongMamba-enhanced Falcon-Mamba, Llama2-7B-chat-4k [b], XGen-7B-8k[c] and Vicuna-v1.5-7B-16k [d] on the long-context understanding tasks from LongBench. To enhance readability, the table is divided into two sections (top and bottom), with all values expressed as percentages.

We can draw the following observations:

1. The vanilla Mamba exhibits lower performance compared to Transformer models on long context understanding tasks. Specifically, Llama2-7B-chat-4k [b], XGen-7B-8k[c], and Vicuna-v1.5-7B-16k [d] achieve 2.4%, 1.2% and 6.7% higher average accuracy, respectively, compared to the vanilla 7B Falcon-Mamba model (refer to the last column in the bottom half of the table).
2. Our LongMamba can enhance the performance of vanilla Mamba models and narrows the gap with or even surpass Transformer models. As shown in the last column of the bottom half of the table, LongMamba achieves a 2.8% increase in average accuracy compared to the vanilla Mamba model, outperforming Llama2-7B-chat-4k (+0.4%) and XGen-7B-8k (+1.6%), and reducing the gap with Vicuna-v1.5-7B-16k from 6.7% to 4.0%.

We believe that this is a positive signal for the research community, implying that Mamba's long-context capability can match, if not surpass, that of Transformers. Our LongMamba approach demonstrates the potential to unlock Mamba's long-context capabilities in a training-free manner, serving as a promising step toward activating these latent abilities and inspiring the development of more sophisticated long-context solutions for Mamba.

We greatly appreciate the insightful comments and the concerns you raised, and have addressed them below:

---

W1 & Q5: LongMamba relies heavily on the DeciMamba framework (specifically, the token filtering idea), raising questions about its independent novelty. Have the authors considered alternative strategies for decay adjustment?

We humbly clarify that the key contribution of LongMamba lies in the insight distinguishing global and local channels in Mamba, which serves as the foundation for enhancing its long-context capabilities. While we use token filtering to implement this insight, we emphasize that it is just one of many possible approaches to leverage or implement our insight, showcasing the broader applicability and originality of our contribution. Specifically, our new contributions beyond DeciMamba are as follows:

1. Key Insight:

We discover that the exponential hidden state decay of the global channels is the primary cause of Mamba’s constrained receptive field and limited performance on long-context tasks. This finding is also highly recognized by other reviewers: Reviewer P417 remarked, "The finding about the categroies of channels of Mamba models (local and global) is interesting and meaningful, which could bring insight to the community for better understanding the Mamba model's behavior", Reviewer 1AHb highlighted, “the discovery and explanation of Mamba models’ limited performance in long contexts are quite interesting," and Reviewer GwXu stated, “the main result is very interesting.”

2. Token Filtering Implementation:

Leveraging this key insight, we propose specially handling the global channels to enlarge their receptive field by applying selective token filtering exclusively to the global channels while preserving the locality of the local channels. This strategy has outperformed DeciMamba and maintained the same language modeling perplexity on contexts 10x longer, further validating the significance and practicality of our key insight. This has also been recognized by other reviews, such as “Overall a simple mechanism that seemingly enables Mamba to generalize to much longer contexts” by Reviewer GwXu.

3. Alternative Implementation:

To further demonstrate the generality of our insight, during the rebuttal period, we implemented and validated an alternative method.

This method dubbed “decay normalization” adjusts the decay by normalizing $\Delta_t$ for all tokens in the same global channel with a scalar constant c (i.e., $\Delta’_t = \Delta_t / c$), and subsequently derives the adjusted $\bar{A}’_t$ and $\bar{B}’_t$ using Eq.7 of the submitted manuscript. The factor $c=c(S)$ is computed to satisfy Eq.16 in Section 5.2 of the submitted manuscript, similar to how we obtain $g(S)$.

We evaluated this alternative decay adjustment method on both the LongBench and PG19 tasks. The results are summarized in Tab. A and Tab. B below, where “Ours: Token Filtering” corresponds to the decay adjustment method described in the submitted manuscript, and “Ours: Normalization” represents the alternative approach proposed above.

Q4: How does LongMamba perform on other LongBench tasks, namely 2WikiMQA, TREC, TriviaQA, LCC, and RepoBench-P?

Thank you for raising this question! The short answer is that LongMamba consistently performs well in these tasks as well. Specifically, we have provided the benchmark results for all LongBench tasks in Tab. A and Tab. D (in our response to you above) for the 1.4B Mamba model and the 1.3B Mamba2 model, respectively. In addition, following your suggestion, the performance on {2WikiMQA, TREC, TriviaQA, LCC, and RepoBench-P} with the 1.4B Mamba model is listed below in Tab. E:

Tab. E: Benchmarking results on LongBench dataset with the 1.4B Mamba model. Note that this table is a subset of Tab. A.

As shown in Tab. E, we can observe that the proposed LongMamba can notably improve the performance of Mamba models across all tasks (the accuracy in the last row is consistently higher than that in the third row), again demonstrating the effectiveness of the proposed method.

---

We are still working on addressing Q3: Would the proposed approach generalize to other SSMs beyond Mamba, including hybrid Transformer/SSM architectures, or is it specifically tailored to Mamba’s architectural nuances? We will post a response to this question before the end of the rebuttal period. (Update: This question has been addressed in “Author Response #5” below.)

W3: The method for handling global channels’ decay distribution adjustment is not fully elaborated. It remains ambiguous whether averaging across global channels fully mitigates decay issues over extended contexts, potentially affecting LongMamba’s consistency in varied contexts (Figure 2c, Decay Distribution Adjustment).

First, we humbly clarify that our method does not perform cross-channel averaging across all the global channels. Instead, we apply token filtering independently for each channel, ensuring that the exponential decay is effectively addressed for all global channels. Specifically, as introduced in Section 5.2 of our submitted manuscript, we first obtain the distribution of $\bar{A_t}$ for each channel by profiling a subset of the training sequences and compute a threshold $g(S)$ for each global channel, which is used for their decay alignment by filtering out tokens where $\Delta_t < g(S)$ before the hidden state update.

In addition, to demonstrate the effectiveness of our algorithm in mitigating the exponential decay issue, we present the per-channel cumulative decay (as defined in Eq. 14) for Mamba models, with and without LongMamba, in Appendix B.2 of the revised manuscript. As illustrated in these figures, LongMamba can significantly reduce the exponential cumulative decay in the global channels and thus extend the effective receptive field. Detailed discussion on this observation is provided in Appendix B.2 of our revised manuscript.

---

Q2: Is a similar exponential decay observed in the Mamba2? Does LongMamba improve the long context understanding with Mamba2?

Thank you for the insightful question! Yes, we have also observed that Mamba2 exhibits a similar exponential decay issue and illustrated the cumulative decay of the Mamba2 model in Appendix B.3 of the revised manuscript.

Following your suggestion, we have also applied LongMamba to Mamba2 and find that LongMamba can notably improve the long-context understanding of Mamba2, with experimental results presented in Tab. C and Tab. D below for the PG19 and LongBench datasets, respectively.

Tab. C: Perplexity on the PG19 dataset with 1.3B Mamba2 models pre-trained on 2k sequence length. This table compares the performance of vanilla Mamba2 model and LongMamba under different test sequence lengths.

Tab.D: Benchmark results on LongBench with the 1.3B Mamba2 model pre-trained on 2k sequence length. This table compares the performance of vanilla Mamba models and LongMamba, under different test sequence lengths. To enhance readability, the table is divided into two sections (top and bottom), with all values expressed as percentages.

Result summary: We can observe that LongMamba also enhances the performance of the Mamba2-1.3B model on both datasets. Specifically, as shown in Tab. C, LongMamba can consistently and significantly reduce perplexity across all evaluated context lengths (which range from 10k to 100k tokens). Meanwhile, Tab. D demonstrates that LongMamba improves the average accuracy on LongBench by 8.2% (refer to the last column of the bottom half of Tab. D).

Tab. A: Benchmarking results on LongBench dataset with 1.4B Mamba models. This table compares the LongBench performance achieved by the vanilla Mamba model, DeciMamba, as well as the two proposed methods that specially handle the global channels stemming from our insight. To enhance readability, the table is divided into two sections (top and bottom), with all values expressed as percentages.

Tab. B: Perplexity on the PG19 dataset with 1.4B Mamba models. This table compares the language modeling performance of DeciMamba as well as the two proposed decay adjustment methods.

As observed in Tab. A and Tab. B, the alternative method (“Ours: Normalization”) can consistently improve Mamba’s performance for long-context tasks. For instance, it brings a 5.4% higher average accuracy on LongBench (as shown in the last column of the bottom half of Tab. A) and consistently reduced perplexity compared to DeciMamba on the PG19 dataset (as shown in the second and last row of Tab. B).

This set of experiments further validates the effectiveness and generality of our insight. We believe this insight could shed light on future research on other decay adjustment methods and innovations to enhance SSMs’ long-context capabilities.

---

W2: The paper does not systematically address threshold selection for distinguishing global versus local channels. This selection is left to empirical values, which may lack generalizability across different datasets or models.

We would like to humbly clarify that the threshold is determined using a systematic strategy for each model, and is then fixed and generalizable across all evaluation tasks for this particular model. In other words, for all experiments throughout the submitted manuscript, we use the same selection threshold (1e-30).

The rationale behind this approach is that we empirically find the optimal threshold generalizes well across tasks. We conjecture that this is because the distinction between global and local channels is not sensitive, as the top global channels might dominate the contributions due to the exponential decay effects. This allows us to rapidly determine the optimal threshold on one task and reuse it across all evaluation tasks for a given model. To demonstrate this, we calibrate the threshold using the Passkey Retrieval task through a grid search (with {1e-20, 1e-25, 1e-30, 1e-35, 1e-40} as the candidates) and reuse the identified optimal value for all other tasks. As shown in Fig. 4 and Tab. 1 of the submitted manuscript, the resulting performance of our method already outperforms vanilla Mamba and DeciMamba, demonstrating the effectiveness of this strategy. Additionally, the results achieved on other tasks, such as those on LongBench and PG19 shown in Tab. A and Tab. B in our response above, also adopt this simple strategy and demonstrate consistent improvements.

Following your suggestion, we have added a clarification for the threshold calibration process in Appendix A of the revised manuscript.

We greatly appreciate your encouraging recognition of our contribution and your constructive questions and suggestions, and have addressed the questions below:

---

Q1: End of Section 5: line 333, I think you meant Eq. 16, not Eq. 18?

Thank you for catching this typo! You are correct, and we have updated the line in the revised manuscript to use “Eq. 16”.

---

Q2: What $\bar{A}'_t$ is if updates are greater than the threshold?

If $\bar{A}’_t$ is greater than the threshold, their values remain unchanged (i.e., the same as $\bar{A_t}$). We have clarified this in line 332 of the revised manuscript.

---

Q3: How is $g(S)$ computed? Is it computed to satisfy Eq. 16?

Yes, you are correct. $g(S)$ is computed to satisfy Eq. 16, aiming to ensure that the cumulative decay on the identified global channels at the test sequence length $S$ aligns with that at the training sequence length $L$ when $S>L$.

To further clarify this, we have reorganized our proposed solution and its corresponding implementation, summarizing them as follows. We will further clarify this in the final version.

1. Our proposed two-step solution, as detailed in Section 5.2 of our submitted manuscript:

Step-1: We first obtain the distribution of $\bar{A_t}$ by profiling on a subset of training sequences.

Step-2: For the target test sequence length $S > L$, we compute a threshold $g(S)$ that satisfies Eq.16 in our submitted manuscript.

2. The detailed implementation:

As outlined in Sec. 6.1 of the submitted manuscript, we sample 10 training sequences to obtain $\bar{A_t}$ distribution in Step-1 and build a lookup table by computing $g(S)$ for every 1,000-token increase in context length (i.e., S={1k, 2k, 3k, …, nk, …}) in Step-2. During inference, the test sequence length $S$ is rounded up to the nearest entry in the lookup table, and the corresponding threshold $g(S)$ is retrieved from the lookup table.

---

Q4: Local/Global channel: Are the numbers 0-300-600-900-1200 just percentiles of a fixed model?

Thank you for asking this. We would like to clarify that the numbers (0-300-600-900-1200) in Fig. 1 of our submitted manuscript represent the channel indices in the model's weight tensor, not any rankings or percentiles. We selected three specific local channels and two global channels as examples to illustrate these concepts in Fig. 1. This has been clarified in the revised manuscript.

---

Q5: Can you include a graph showing the cumulative graph of receptive fields for each channel? Especially this would be interesting for training sequence length and a much longer sequence length.

Thank you for the suggestion! We have added the suggested figure in the revised Appendix B.1. From the figure, it can be observed that for global channels: (1) when the sequence length is smaller than the training sequence length, the receptive field grows linearly with increasing sequence length, and (2) beyond the training sequence length, the receptive field no longer increases linearly and sometimes even decreases, limiting the model’s ability to capture global information from longer sequences.

Summary of All Responses:

Added Visualizations:

1. Receptive fields under different context lengths

   • Added in Appendix B.1 of the revised manuscript.

2. Effect of LongMamba on mitigating the exponential hidden state decay

   • Included in Appendix B.2 of the revised manuscript.

3. Hidden state decay factor in Mamba-2

   • Provided in Appendix B.3 of the revised manuscript.

Added Experiments:

1. Proposing and validating an alternative method to implement our key insight

   • Results in the response to Reviewer xMy1’s W1 & Q5.

2. Extending LongMamba to Mamba-2

   • Results in the response to Reviewer xMy1’s Q2.

3. Performance on additional LongBench tasks

   • Full benchmark results in Tab. A of the author response #2 to Reviewer xMy1.

4. Applying LongMamba to hybrid Transformer/SSM architectures

   • Results in the response to Reviewer xMy1’s Q3.

5. Comparison with Transformer models

   • Results in the response to Reviewer P417’s W1.

6. Measuring the compute and memory overhead of LongMamba

   • Results in the response to Reviewer P417’s Q3.

7. Extending LongMamba to larger Mamba models

   • Results in the response to Reviewer P417’s Q3.

8. Benchmarking on the RULER dataset

   • Results in the response to Reviewer 1AHb’s Q3.

Clarifications and Fixed Writings:

1. Discussion of our work’s novelty and differences from previous works

   • Included in the response to Reviewer xMy1’s W1 & Q5.

2. Systematic threshold selection process

   • Detailed in Appendix A of the revised manuscript.

3. Algorithm for decay adjustment

   • Explained in the response to Reviewer xMy1’s W3.

4. Miscited equation in Sec. 5, Line 333

   • Corrected in the revised manuscript.

5. Clarification on \bar{A’_t} updates

   • Provided in Sec. 5.2 of the revised manuscript.

6. Derivation of g(S)

   • Algorithm detailed in the response to Reviewer GwXu’s Q3.

7. Channel index definition in Fig. 1

   • Updated in the caption of Fig. 1 in the revised manuscript.

---

[a] Ben-Kish, Assaf, et al. "DeciMamba: Exploring the Length Extrapolation Potential of Mamba." arXiv preprint arXiv:2406.14528 (2024)

[b] Dao, Tri, et al. “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality.” arXiv preprint arXiv:2405.21060 (2024)

[c] Dong, Xin, et al. “Hymba: A Hybrid-head Architecture for Small Language Models.” arXiv preprint arXiv:2411.13676. (2024)