Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity

Common Response

We refer the reviewer to the Common Response in the Rebuttal to WpKK.

Individual Response

We thank the reviewer once again for their positive feedback, and remain open to include any suggestion for improvement, or answer any question they might have.

Common Response

We refer the reviewer to the Common Response in the Rebuttal to WpKK.

Individual Response

SD4 is mentioned without definition in abstract (and the 1st page).
We thank the reviewer for pointing out the mistake: this has been corrected in the text. We further thank the reviewer for their favorable review, and remain ready to address any question or suggestion for improvement they may have.

Common Response

We are grateful for the positive comments from reviewers on our paper, particularly regarding its clarity (kLHf: “this paper is of very high quality”; R4Me: “quite clearly written”), soundness (kLHf: “very refreshing yet solid”; R4Me: “These results are proven rigorously”), and impact (kLHf: “I believe this work is significant. The theory echos closely with practice and provides practically useful insights for practitioners”; R4Me: “a useful addition to such literature”; WpKK: “this work provides a valuable perspective in understanding a core design element...and might inspire follow-up work”).

We answer the remaining questions from the reviewers in the Individual Response section. Additionally, if the reviewers have any other remarks or recommendations that can help further improve the quality of the paper, we remain at their disposal to address them.

Individual Response

Does Mamba2 fit into the function approximation analysis?
Indeed it does, and the reviewer is right in that it is worth highlighting more clearly. We do briefly mention (Sec3, after Eq(7)) how the Mamba2 layer is a simplification of Mamba that prescribes a state matrix parameterized by a single scalar $\boldsymbol{\Lambda}=\lambda \boldsymbol{I}$. Substituting this into (8) would remove the dependency of $\lambda$ from the hidden-state component $n$, that is we would have $\lambda_n \equiv \lambda$ for all $n=1, \ldots, N$ in the integral. Nonetheless, for the proof in Sec4.1, it suffices to set $\lambda_n = -1$ for all $n$ (see Line 605), so the proof still holds even in the Mamba2 framework. A similar reasoning also applies to Sec4.2, where again for Mamba2 $\lambda_n \equiv \lambda$ would be constant over $n=1, \ldots, N$, but it still does not affect the overall derivation. Following the reviewer's remark, in the main text we have added a note on the validity of both Thm1 and Lem1,2, for Mamba2.

Does the analysis shed light on the expressivity differences between Mamba1 and Mamba2?
The analysis in Sec4 (see also the response above) highlights that, from the point of view of function expressivity alone, if we consider only the SSM-layer, then there is no expressivity difference between Mamba and Mamba2 on approximating Haar Wavelets: Mamba2's simplification of setting $\lambda_n \equiv \lambda$ across state dimension does not hinder this ability.
Nonetheless, the performance difference between Mamba and Mamba2 has been clearly shown empirically. Motivated by this, we extended our analysis to the whole architecture in Sec5. Our results in Thm2,3 highlight that, thanks to its per-SSM-parameter short-convolution layers, Mamba2 can recover more parameter-efficient solutions than Mamba for the MQAR synthetic task, hinting at a possible explanation for its superior performance. We believe analyzing other, more complex tasks to be an interesting next step in shedding more light on the expressivity differences between Mamba and Mamba2.
We thank the reviewer for raising this point. We took this chance to update the manuscript to highlight more clearly the implications that Thm2 and Thm3 have on the relative parameter-efficiency of Mamba and Mamba2 in solving MQAR.