Retentive Network

Thanks for your positive comments.

Q1: Can you provide more insights into why RetNet starts to outperform Transformer when the model size is larger than 2B?

A1: Because of the recurrence nature of the proposed method, the dimension of "hidden states" is critical for model performance, which is similar to the concept of LSTM hidden size. We find that expanding the width is beneficial for retention, especially for smaller size. Along with model size increases, the hidden size becomes larger.

---

Q2: Have you tried different learning rates to investigate their impact on the scaling of the RetNet model? If so, what were the results and how did they affect the model's performance and scalability?

A2: We follow the learning rate schedule used by Transformers for fair comparisons in the paper. The suggested investigation would be valuable to obtain the relation between optimal learning rate and model size.

---

Q3: Use of $\gamma$? How is $\gamma$ determined?

A3: As descibed in Line 66 and Eq. (3), the decay term $\gamma$ is introduced by diagonalizing the matrix $A$. As shown in Table 3, the ablation "without $\gamma$ decay" performs worse than the proposed retention, indicating the effectiveness of our design. The physical meaning of $\gamma$ is relative position bias, such as Alibi[1], and xPos[2]. Following the above works, we assign different decay values for the heads. Moreover, the decay speed is similar with previous position embedding works, rather than heuristic search of $\gamma$.

[1] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

[2] A Length-Extrapolatable Transformer

We hope the above explanation clarifies the rationale behind our designs.

Q1: The authors introduce a new term called "Retention," but this is essentially the same as Linear Attention without the denominator, which has already been proposed in The devil in linear transformer.

A1: The term is proposed to avoid confusion with the pioneer work "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention", where linear attention is with the denominator. We can add the discussion of The devil in linear transformer (Transnormer) in the paper. We also list several key differences as follows.

• We derive the design starting from the recurrent view, rather than empericially modifying the original attention.

• The theoretical derivations naturally have position embedding and the decay term ($\gamma$), which is critical for stable training and good performance. As shown in Table 3, the ablation "without $\gamma$ decay" performs worse than the proposed retention. Directly using the QKV implementation can not easily converge as expected at larger scale. The Transnormer model additionally interleaves diagonal blocked sparse softmax attention.

• The three equivalent computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent representations, are also important features of retention.

• Although the overall forms seem similar, but the specific equations are still different. More importantly, the modeling and training behaviors are very different, based on the theoretical nature of the proposed method.

We appreciate that you point out this issue. We are open to discuss these key elements and discuss Transnormer in the paper.

---

Q2: Lack of comparison with the baselines on open source pretraining data. All the training experiments are conducted on in-house data mixtures, which harms the reproducibility.

A2: The training corpora used in the paper are all public available. They can be easily downloaded online for acadamic purpose. We use the same training data across models for fair comparisons.

---

Q3: The paper doesn't compare RetNet with other linear attention model (such as GLA, RWKV, Mamba) on downstream tasks with standard metrics instead of perplexity. Table 2 only include RetNet and Transformer.

A3:

• As shown in Table 1, we compare RetNet with Hyena, RWKV, Mamba, H3 on fine-grained language modeling evaluation and MMLU answer perplexity. The fine-grained language modeling evaluation correlates well with downstream tasks.

• As pointed out in [1][2], robustly evaluating accuracy metrics need larger model size. Otherwise the accuracy metrics tend to be affected by the "emergence" issue. Transformer is still a quite strong architecture, so we compare with Transformer with 6.7B model size in Table 2, rather than scaling up all the variants to 6.7B for robust comparisons. The metric protocal is by design for more scientic evaluation.

We hope the above explanation clarifies the rationale behind our experiment designs.

[1] Are Emergent Abilities of Large Language Models a Mirage?

[2] Understanding Emergent Abilities of Language Models from the Loss Perspective

---

Q4: The efficiency measurment of RetNet+ is absent.

A4: The efficiency measurment of RetNet+ can be obtained by interpolating the efficiency results of hybrid blocks, depending on the ratio of the mixed blocks, which are provided in the main content.

---

Q5: The evaluation on MMLU/Qasper is using perplexity but not the widely-used accuracy/F1 metric. The perplexity results don't necessarily mean that the model can make correct choices for the samples in MMLU, and has less guidance for the model's downstream performance.

A5: The metric protocal is by design for more scientic evaluation. As pointed out in [1][2], robustly evaluating accuracy metrics need larger model size. Otherwise the accuracy metrics tend to be affected by the "emergence" issue. In comparison, we report accuracy numbers for scaling-up settings, such as Table 2 (i.e., scaling up size) and Table 5 (i.e., scaling up training tokens and size).

[1] Are Emergent Abilities of Large Language Models a Mirage?

[2] Understanding Emergent Abilities of Language Models from the Loss Perspective

---

Q6: Missing citations: The authors should also cite [1] for the normalization after retention, and discuss the details of the triton implementation of RetNet and its difference from the implementation in the Flash Linear Attention [2] library.

A6: We can add them in the camera-ready version.

Thank you for the positive review.

Q1: The inference results in Figure 5 start with 2048. What's the inference speed for shorter sequences?

A1: The RetNet inference speed remains almost constant across length, and Transformers' speed can be extrapolated according to Fig 5. As shown in Figure 5(c), we also compare inference latency of Transformer with 1024 length (yellow). It shows there is still improvement for sequences shorter than 2048.

---

Q2: The claim that "None of the previous work can achieve strong performance and efficient inference at the same time compared with Transformers" is overly strong and potentially misleading. Recent advancements in efficient modeling, such as Mamba, have demonstrated better scaling properties than Transformers.

A2: We can improve this part as suggested to take more recent advances into consideration.

---

Q3: It would be great to include the training loss curve for both Transformer and RetNet.

A3: We saved the loss curves in Tensorboard. We can provide them for future research work.