Q1. Theoretical analysis of different decay parameterizations

R1. We provide visualizations of the decay median in Figures 1 and 2 of the paper. From a theoretical standpoint, the decay mechanism functions analogously to a soft sliding window, influencing the model’s effective receptive field. Specifically, a decay value close to 0 results in an overly narrow receptive field, which is suboptimal for sequence modeling. Conversely, a decay value close to 1 fails to enforce sufficient locality, thereby limiting the model’s ability to effectively capture language-specific patterns using linear attention mechanisms.

Q2. Computational cost overhead of different decay parameterization schemes

R2. we report the running speeds under different parameterization schemes using the TGS (tokens per GPU per second) metric. As shown, all methods—except for LightNet—exhibit comparable TGS values.

This discrepancy is attributable to the computational overhead introduced by LightNet, which requires the use of the logcumsumexp operation, as detailed in Table 1. In contrast, the other methods involve only element-wise operations, leading to more efficient computation and similar runtime performance.

Q6. The relationship between this paper's framework and Mamba2's framework

R6. Thank you for your suggestion. We provide the correspondence between several common Linear Attention models and Mamba in the table below. Specifically, Query in Linear Attention corresponds to C in SSM, Key corresponds to B, Value corresponds to X, and Decay corresponds to A.

Q7. Supplementing DPLR-related literature

R7. Regarding the DPLR section, we will supplement S4 as a reference;

Q8. Optimizing table visualization

R8. Thank you for your suggestion. We will optimize this in subsequent versions.

Q9. How Simple Decay simplifies the design

R9. See R3.

Q5. Adding Transformer Baseline

R5. Thank you for your suggestion. We have added the Transformer Baseline (LLaMA architecture), with results as follows:

We can see that: LLaMA demonstrates superior performance on perplexity (PPL) metrics, and most linear attention models tend to underperform relative to LLaMA in this regard. However, on multiple-choice tasks, linear attention models achieve performance that is comparable to LLaMA.

Q3. Which parts the simplification is based on.

R3. Thank you for your suggestion. We derived the final Simple Decay mainly based on the following analysis:

• Section 5.1 analyzes the advantages and disadvantages of various parameterized decay schemes. We found that decay performs best when the median is around 0.8. Based on this, we designed simple decay to ensure that the decay median in the early stages of network training is at a value relatively close to 1;
• Section 5.2 indicates that parameter sharing cannot be used arbitrarily, and its effect is equivalent to low rank decay, so we used low rank decay in the final Simple Decay scheme;
• Section 5.4 indicates that in most cases, using Decay no longer requires RPE, so our Simple decay also does not use RPE;

Q4. Comparison with Mamba2

R4. Our goal is not to propose a decay scheme better than Mamba2, but to analyze the design of decay and provide a simpler scheme with comparable effectiveness. We list the implementation methods of Mamba Decay and Simple Decay below: Mamba2:

# init
dt = torch.exp(torch.rand(self.decay_dim,) * (math.log(dt_max) - math.log(dt_min)) + math.log(dt_min))
dt = torch.clamp(dt, min=dt_init_floor)
# Inverse of softplus: https://github.com/pytorch/pytorch/issues/72759
inv_dt = dt + torch.log(-torch.expm1(-dt))  # expm1(x) = exp(x) - 1
A = torch.empty(self.decay_dim, dtype=torch.float32).uniform_(
    *A_init_range
)
self.dt_bias = nn.Parameter(inv_dt, requires_grad=True)
self.log_A = nn.Parameter(log_A, requires_grad=True)

# compute
k = F.softplus(self.k_proj(x) + self.dt_bias)
log_f = -self.log_A.float().exp() * k

Simple Decay:

# take x = 0 as median, 1 / (1 + exp(-(median + delta))) = a => 1 + exp(-delta) = 1 / a => exp(-delta) = (1 / a - 1) -> exp(delta) = a / (1 - a) => delta = log(a / (1 - a))
a = self.threshold
delta = torch.ones(self.decay_dim) * math.log(a / (1 - a))
self.delta = nn.Parameter(delta, requires_grad=True)
  
# compute
log_f = F.logsigmoid(self.k_proj(x) + self.delta)

As can be seen, Simple Decay's initialization and computation parts are more concise and easier to understand, with fewer hyperparameters, and the final effect is comparable to or better than Mamba2.

Q1. Experiments are much narrower than what is in the title.

R1. Thank you for your suggestions. First, SSM, Linear Attention, and Linear RNN are essentially different formulations of the same model. We list their mapping relationships in the table below, and we uniformly use Linear Attention to describe them in the paper:

So far, the mainstream Linear Attention models still primarily use diagonal matrices for state transition, such as Mamba1/2, GLA, HGRN2, etc.

On the other hand, the diagonal part of DPLR is also a type of decay. We will add discussion of this part in subsequent versions and have supplemented experiments (see R2).

Q2. Whether the conclusions apply to DPLR.

R2. Thank you for your suggestion. In response, we have conducted additional experiments to evaluate the performance of Simple Decay within the DPLR scenario, specifically comparing scalar decay and vector decay under a consistent parameterization scheme.

The results indicate the following:

• In DPLR settings, scalar decay consistently underperforms relative to vector decay when evaluated under the same parameterization.
• Simple Decay achieves superior average perplexity (PPL) and accuracy across nearly all DPLR scenarios.

These findings further support the effectiveness and robustness of Simple Decay and highlight the advantages of vector decay in this context.

Dear reviewer, did our response address your concerns? If you have any other questions, please feel free to ask.

Q1. Limited novelty for the Simple Decay

R1. Thank you for your insightful comments. In response, we would like to emphasize that the primary contribution of this work lies in the systematic exploration of the design space of decay mechanisms within linear complexity models, rather than the proposal of a novel decay strategy per se. The Simple Decay method is presented as an illustrative example, derived from our empirical observations. Notably, this method demonstrates both greater simplicity and improved performance compared to existing approaches.

To the best of our knowledge, prior studies have not comprehensively examined this design space, particularly with respect to the following four dimensions:

(1) parameterization strategies,

(2) the influence of parameter sharing,

(3) decay granularity, and

(4) compatibility with relative positional encoding.

We believe that the insights gained from this investigation offer meaningful contributions to the understanding and development of decay mechanisms in this domain.

Furthermore, to assess the generalizability of Simple Decay, we have evaluated its performance under the DPLR framework (including both scalar and vector decay settings), as presented in the following table:

As can be seen, simple decay achieves better average PPL and accuracy in almost all DPLR scenarios, demonstrating the generalizability of our design.

Q2. Several conclusions are not entirely surprising insights.

R2.

Thank you for your thoughtful suggestion. We would like to clarify that, under the same parameterization scheme, scalar decay generally underperforms compared to vector decay—a distinction that, to our knowledge, has not been systematically addressed in the existing literature. For instance, while Mamba2 outperforms Mamba1, it employs scalar decay, whereas Mamba1 uses vector decay. A direct comparison between the two models might therefore lead to misleading or contradictory conclusions.

Importantly, the performance differences between Mamba1 and Mamba2 cannot be attributed solely to their respective decay strategies, as the models differ in several other aspects, such as state size. Consequently, attributing the performance gains exclusively to the choice of decay type risks confounding variables.

Our work aims to fill this gap by systematically disentangling the effects of individual design choices. Through carefully controlled experiments, we investigate which factors most significantly influence decay performance and identify which architectural components may be unnecessary.

Dear reviewer, did our response address your concerns? If you have any other questions, please feel free to ask.

Dear reviewer, there are less than 24 hours left until the review deadline. If you have any questions, please let us know.