Liger: Linearizing Large Language Models to Gated Recurrent Structures

Dear Reviewer 5t3M,

Thanks for your thorough reviews and we appreciate the attention to detail and would fix the writing errors in the revision. We give point-by-point responses to your comments based on your questions.

Q1: Is pooling performed across the time dimension?

No, the pooling operation is performed accross hidden dimension. We perform linearization by pooling the model's $K \in \mathbb{R}^{T \times D}$ into $G \in \mathbb{R}^{T \times D_G}$, where $D_G$ depends on the gating module of the specific model used, such as $G \in \mathbb{R}^{T \times S}$ in GSA, where $S$ is the number of memory slots. This form can achieve the effect of dynamic gating without introducing any learnable parameters during the linearization process. It is simple and has been demonstrated to be particularly effective in experiments.

Q2: 'GLA denotes Gated Recurrent Modeling' -> should GLA be GRM instead?

Yes, thanks for your careful review. Here GLA should be GRM instead. GRM represents a general form of various gated recurrent modeling methods and their variants, and GLA is an instance of GRM.

Q3: What does Liger-GLA mean? Does this not include Sliding window attention?

Liger-GLA means using Liger proposed in our paper for linearization of LLM to gated recurrent structures, which adopts Liger Attention, a intra-layer hybrid attention including Sliding Window Attention and using GLA as the instance of GRM.

Q4: What do learnable feature map modules mean?

The original softmax attention is calculated as $O=\text{softmax}(QK)V$, and its complexity is $\mathbb{O}(T^2)$ quadratic over the length of the sequence. Linear attention is achieved by removing the softmax operation and replacing it with feature map modules $\phi$ acting on $Q$ and $K$ and reducing the computational complexity to $\mathbb{O}(T)$ through the right multiplication technique. The specific formula is $O=\phi(Q)\phi(K)V$, so that the product of $\phi(Q)\phi(K)$ can approximate the output of softmax attention. For example, [1] setting feature map to $\phi(x)=\text{elu}(x)+1$ introduces non-negativity similar to softmax attentioin to achieve better performance.

Learnable feature map modules introduce learnable parameter for $\phi$ optimization, which makes the model adaptive to data and improves expressiveness to better approximate the effect of softmax attention. For example, the feature map module design in [2] adopts: $\phi(x)=\exp(\mathbf{W}^\top x+b)$.

In our proposed Liger, we set $\phi(x)$ which simply uses the normalization function, which can ensure that the distribution of the product of $\phi(Q)\phi(K)$ is still non-negative and approximates the original $\text{softmax}(QK)$ output distribution, and there is no need to introduce any learnable parameters, which is very helpful for LLM linearization and demonstrated effective in our experiments.

---

[1] Angelos Katharopoulos, et al. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. ICML, 2020.
[2] Michael Zhang, et al. The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. ICLR, 2024.

Dear Reviewer 2WYJ,

We are truly grateful for the time you have taken to review our paper and your insightful comment. We address your questions in the following.

Q1: Discussion on intra-layer hybrid attention with similar designs

We carefully read these works for further comparison and discussion.

Hymba uses a intra-layer hybrid attention which combines softmax attention and SSM via independent parameterizations: $o_t=\beta_1\text{norm}(M_{attn}x_t)+\beta_2\text{norm}(M_{ssm}x_t)$, where $M_{attn}$ and $M_{ssm}$ employ separate projection matrices $W_Q/W_K/W_V$ vs. $W_A/W_B/W_C$. Liger Attention eliminates this duplication by: $o_t=\alpha\text{GRM}(q_t,k_t,v_t)+\beta\text{SWA}(q_t,k_t,v_t)$, which shares matrices between sliding window attention (SWA) and gated recurrent modules (GRM), avoiding the need for introducing extra learnable parameters. This parameter efficiency directly supports better linearization while retaining generality—GRM subsumes (but is not limited to) SSM.

Titans' hybrid form also similarly uses independent parameters to construct linear and attention modules respectively, and is essentially an inter-layer hybrid rather than intra-layer, because the output of its memory module (linear module) will be used as the input of the attentnion module, and vice versa.

We would respectfully cite and supplement the discussion of related works.

Q2: Significant degradation in performance on MMLU

We have noticed that most linearization methods have significant degradation on MMLU. We hypothesize that the performance gap partially stems from MMLU's evaluation design, which assesses both knowledge recall in pretrained weights and context-based answer retrieval through its 5-shot multiple-choice format[4]. While this setup requires models to identify correct answer indices from contextual prompts, prior research demonstrates that linear recurrent architectures exhibit inherent limitations in such retrieval tasks[5].

Despite this, we still try our best to alleviate this problem. We linearize Liger from Llama-3-8B-instruct with stronger instruction following ability. Besides, we investigate whether we can improve MMLU performance from stronger LLM (e.g. Qwen2.5).

Although there is still a significant drop on MMLU, we find that Liger can benefit from the instruct model to improve 5-shot performance, and can also benefit from more powerful model (Qwen2.5) for such task (only 1.6 lower than Mamba2-Llama3-8B 50% Attn), further narrowing the gap with the original LLM.

Q3: Why does the gate need to be derived from the Key instead of other elements

The key matrix usually serves as memory indexing in transformer or linear model[1], which is similar to the gate that determines which part of the memory should be retrieved or forgotten. In this view, it is intuitive to choose Key for gate construction. MetaLA[2] also points out that the linear recurrent model needs to meet the "least parameter approximation" condition to achieve the optimal linear attention design. In this case, the key matrix is ​​redundant and can be used to construct the gate, which also provides motivation and theoretical support for gate derived from Key.

We have supplemented experimental comparisons of gate derived from Q and V:

We can observe that the effect of constructing the gate module from any Q, K, V is ​​significantly better than randomly initialized gate, and the overall performance of gate derived from Key is ​​more outstanding, which is consistent with the conclusion in [2][3] that the parameters of the Key Matrix are more redundant and can act as gating mechanism.

Q4: How much do the LoRA weights change

LoRA is only used to finetune QKV that only 0.059% of total weights need to be trained. Let $W$ be the sum of each weight parameter of LLM, we calculate the change in Liger's LoRA weights compared with LoLCATs:

We observe that Liger has a smaller number of changing parameters than LoLCATs, which makes Liger less pretrained knowledge destruction, more efficient and effective.

---

[1] Gershman S. J., et al. Key-value memory in the brain. arXiv, 2025
[2] Chou Y., et al. MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map. NeurIPS, 2024
[3] Qin Z., et al. HGRN2: Gated Linear RNNs with State Expansion. COLM, 2024
[4] Hendrycks D., et al. Measuring Massive Multitask Language Understanding. arXiv, 2020
[5] Waleffe R., et al. An Empirical Study of Mamba-based Language Models. arXiv, 2024

The following are multiple choice questions (with answers) about anatomy.

What is the embryological origin of the hyoid bone?
A. The first pharyngeal arch
B. The first and second pharyngeal arches
C. The second pharyngeal arch
D. The second and third pharyngeal arches
Answer: D

Which of these branches of the trigeminal nerve contain somatic motor processes?
A. The supraorbital nerve
B. The infraorbital nerve
C. The mental nerve
D. None of the above
Answer: D

The pleura
A. have no sensory innervation.
B. are separated by a 2 mm space.
C. extend into the neck.
D. are composed of respiratory epithelium.
Answer: C

In Angle's Class II Div 2 occlusion there is
A. excess overbite of the upper lateral incisors.
B. negative overjet of the upper central incisors.
C. excess overjet of the upper lateral incisors.
D. excess overjet of the upper central incisors.
Answer: C

Which of the following is the body cavity that contains the pituitary gland?
A. Abdominal
B. Cranial
C. Pleural
D. Spinal
Answer: B

Which of the following statements is true of the pupillary light reflex?
A. Its efferent limb is carried in the optic nerve
B. It is mediated by the inferior colliculi in the midbrain
C. It is a consensual reflex
D. Its afferent limb is carried in the oculomotor nerve
Answer:

Dear Reviewer xf9Q,

We sincerely thank you for your careful comments and thorough understanding of our paper! Here we give point-by-point responses to your comments and questions.

Q1: The baseline that only uses sliding window attention is missing.

Thanks for your careful comment. We have supplemented the experiment with sliding window attention (SWA) only.

Notably, whether SWA only or GLA only is used, the performance will decrease, while Liger Attention achieves the best performance.

Q2: It's unclear whether gating is still necessary in this distillation setting

While transformer-based LLMs suffer from computational inefficiency due to unbounded historical storage in the KV-cache, linear recurrent models face challenges in dynamically "forgetting" obsolete information without gating mechanism. Prior work in linear recurrent models [1-4] underscores the critical role of gating in enabling adaptive forgetting, suggesting its necessity even in linearization or distillation setting.

However, existing linearization methods often omit gating or introduce randomly initialized gating modules that need extra training, which incur added training costs and suboptimal performance (e.g., Table 6 shows performance drops when learnable gates are added). In contrast, Liger incorporates gating and optimizes its construction without any parameters introduction and achieves better performance, which proves that gating is necessary for linearization.

It is worth noting that although most linearization methods are based on distillation setting and adopt multi-stage training with further fine-tuning. In fact, our proposed Liger does not adopt any distillation setting. It uses only single stage of LoRA fine-tuning to complete the adaptation of the architecture conversion with only 0.059% of the parameters need to be fine-tuned on 0.02B training tokens and restore over 93% of the LLM performance. The linearization cost significant reduce and it can work on various linear recurrent architectures with gating mechanism, which reflects the simplicity and versatility of Liger.

Q3: Hopefully more discussion on the parameterization of G

The parametric way for $G$ construction is introducing extra learnable parameters, which dynamically adjusts the flow of information through learnable weights and biases (e.g. MLP projection). It is data-adaptive but requires more computing resources and training data compared with the non-parametric way. In the scenario of LLM linearization, we seek to obtain a powerful linear LLM faster at a lower cost under resource constraints.

Our ablation study (Table 6 in our manuscript) show the following results, where Gate Proj refers to construct the gate projection module in parameteric way by introducing extra parameters, which is implemented as the low-rank MLP used in GLA[1].

The results reveal that Liger can achieve better performance by using a simple pooling operation in non-parameteric way to construct the gating module, which is consistent with the concept discussed above about parameterization of $G$.

Q4: Provide details of the pooling operation along the resulting shape of G_t

The form of $G_t$ depends on the specific gated recurrent model employed. Most gated linear models adopt a dynamic time-varying vector gate instead of a simple scalar. In the specific implementation, our pooling operation uses torch.AdaptiveAvgPool1d over the $D$ dimension, which can construct a specified gate form through average pooling or interpolation without introducing any learnable parameters, benefiting the linearization process. For example, the gate of GLA: $G_t \in \mathbb{R}^{D}$ is a vector, while the gate of GSA is $G_t \in \mathbb{R}^{M}$, where $M$ is the number of slots. We can adopt the simple pooling operation to transform the shape from the LLM weight matrix to construct the corresponding gate vector.

---

[1] Songlin Yang, et al. Gated Linear Attention Transformers with Hardware-Efficient Training. ICML, 2024.
[2] Zhen Qin, et al. HGRN2: Gated Linear RNNs with State Expansion. COLM, 2024.
[3] Yutao Sun, et al. Retentive Network: A Successor to Transformer for Large Language Models. arXiv, 2023.
[4] Yuhong Chou, et al. MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map. NeurIPS, 2024.