HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization

Thank you for your thoughtful feedback. We appreciate that Reviewer zMv1 values our theoretical analysis. The concerns raised are mostly due to misunderstandings. We have addressed all concerns below.

---

Q1. What is the computational overhead introduced by HybridNorm? How does this overhead scale with model size?

A1. The direct contribution of RMSNorm to the overall parameter count and computational cost of a large Transformer is, in fact, negligible. Below, we provide a detailed analysis to support this claim.

For simplicity, we consider only the MHA and the SwiGLU activation function, excluding the Embedding and Output layers. Suppose the hidden dimension is $d$, the intermediate FFN size is $\frac{8}{3}d$, the number of layers is $L$, and sequence length is $s$. As shown in the table below, RMSNorm constitutes only a negligible fraction of the actual runtime and memory consumption in Transformer under both Pre-Norm and HybridNorm configurations. And the ratio decreases as the model size increases. Moreover, when employing GQA, the relative overhead of RMSNorm in HybridNorm is further diminished.

Parameters. Each RMSNorm layer introduces $d$ parameters. In a Pre-Norm architecture, there are two RMSNorm layers per Transformer layer, resulting in a total of $2dL$ parameters. In HybridNorm, there are four RMSNorm layers per Transformer layer, yielding $4dL$ parameters in total. Each attention block has four weight matrices (for Q, K, V, and O projections), thereby contributing $4d^2L$ parameters in total for the MHA component. Similarly, the FFN component introduces an additional $8d^2L$ parameters.

Computation. The FLOPs for RMSNorm to process a single token vector of dimension $d$ is $(4d + 4)$. Since the constant term becomes negligible for any reasonably large $d$, this can be simplified to $4d$. Accordingly, for Pre-Norm, the $2L$ RMSNorm operations incur a total cost of approximately $8sdL$, whereas HybridNorm incurs $16sdL$. In practice, however, the dominant computational overhead in a Transformer stems from matrix-vector multiplications, primarily within MHA and FFN modules. Specifically, the FLOPs for MHA amount to $(8sd^2 + 4s^2d)L$, while the FLOPs for FFN total $16sd^2L$.

---

Q2. Advanced schemes like DeepNorm and Sandwich-LN are not evaluated.

A2. We have extended Table 4 to include DeepNorm, Sandwich-LN, and OutputNorm (OLMo-2), as shown below.

We observe that HybridNorm consistently outperforms all other normalization strategies, including the aforementioned advanced methods, across all metrics. This highlights the strength of our approach in both generative perplexity tasks and HellaSwag.

---

Q3. Provide evidence that the observed stability gains generalize to deeper architectures.

A3. In fact, we have already conducted experiments on deeper models in Section 5.5 (Deeper Models), where we evaluate HybridNorm on Transformers with depths ranging from 16 to 29 layers under a comparable parameter budget. As shown in Table 5, both HybridNorm and HybridNorm* consistently outperform Pre-Norm and Post-Norm across all depths. Notably, Post-Norm fails to converge at 29 layers, while our method remains stable and achieves the best performance—demonstrating its superior scalability and training stability in deeper configurations.

To further validate the generalization of our method to large-scale, practically relevant models, we now provide additional results on a 7B dense model with 32 layers, closely following the configuration in [2]. It is important to note that the depth of our model is aligned with that of LLaMA3 and is comparable to the depth of other mainstream open-source models. Therefore, our model should not be considered shallow by any means. The results are summarized below.

Training and Validation Performance:

Downstream Task Performance:

---

Q4. Table 3 shows Pre-Norm and HybridNorm have different sensitivities to initialization methods. What drives this divergence, and can the author provide an intuitive explanation?

A4. For this issue, we attempt to provide a preliminary explanation here. In the Pre-Norm setup, the hidden state update follows the form $h_{t+1} = h_t + f(h_t)$. As the network depth increases, the norm of $h_t$ tends to grow, which leads to the residual branch $h_t$ increasingly dominating over the transformed signal $f(h_t)$. In contrast, HybridNorm applies Post-Norm specifically to FFN, which helps to keep the norm of $h_t$ relatively stable across layers. As a result, the balance between $h_t$ and $f(h_t)$ remains more consistent during training. When using Megatron initialization, which scales down $f(h_t)$ by a factor of $1/\sqrt{2L}$, the relative contribution of $f(h_t)$ diminishes. This shifts the update dynamics closer to that of Pre-Norm, where $h_t$ dominates. As a result, HybridNorm performs better when combined with Megatron initialization.

---

Q5. Limitation concerning the absence of contextual baselines.

A5. Below, we provide clarifications and additional context regarding the positioning and contributions of our work.

Comparison with widely recognized foundation models. We agree that direct comparisons with widely recognized foundation models (e.g., LLaMA 3.2, Qwen2.5) would offer valuable context. However, we would like to clarify that such comparisons are not straightforward due to substantial differences in training setups—most notably, the total number of training tokens. Our HybridNorm models were trained on only 1T tokens, while models like LLaMA 3.2 use up to 15T tokens, often combined with significantly higher-quality data, longer training schedules, and proprietary optimization pipelines. Under such vastly different conditions, direct performance comparisons would not be meaningful and could obscure the architectural contribution of HybridNorm. Instead, we chose to perform controlled, apples-to-apples comparisons between Pre-Norm and HybridNorm under identical architecture, model size, token budget, and training setup, ensuring that any observed improvements can be directly attributed to the normalization strategy. Notably, the Pre-Norm baseline corresponds to the configuration used in LLaMA 3.2. Since LLaMA 3.2 was trained on proprietary data, we conducted our comparisons using the same data and token budget across all settings to ensure fairness.

Orthogonality to data and scale improvements. HybridNorm complements efforts to scale model size and data. As an architectural innovation, it can be seamlessly applied to any Transformer, regardless of scale or training corpus. Our extended experiments show consistent performance gains even under longer training, demonstrating strong scalability and potential for further benefits in larger, data-rich settings. Scaling laws suggest that modest performance gains often demand massive data increases—an expensive and unsustainable path. In contrast, HybridNorm offers meaningful improvements with negligible training overhead, enabling faster convergence and reduced compute cost. Given the limits of high-quality data, architectural enhancements like HybridNorm that shift the slope or intercept of the scaling law provide a valuable and efficient alternative.

Contribution and research focus. Our focus is on understanding and improving the architectural design of normalization in transformers, an area that remains fundamental to the training dynamics, stability, and efficiency of large-scale models. While data and scale are undeniably critical, structure remains the foundation of model design, and our work contributes toward a better understanding of this foundational aspect.

---

Hope our explanations above could ease your concerns. Please feel free to let us know if further clarification is needed.

Thanks for your careful reading and insightful comments. We appreciate that Reviewer Cq1L recognizes the importance of improved normalization techniques for stable transformer training and acknowledges the empirical strength and theoretical soundness of our proposed method. We address your concerns below.

---

Q1. The method, primarily an exploration of normalization placement, has limited novelty.

A1. First and foremost, we would like to emphasize that our method, HybridNorm, is a simple yet effective approach to enhancing both model performance and training stability. The core idea is to integrate Pre-Norm and Post-Norm within each Transformer layer, to balance training stability and model performance. Moreover, HybridNorm* introduces a specialized treatment for a specific layer.HybridNorm adopts a hybrid strategy within each transformer block, leading to more stable gradient propagation during backpropagation and effectively mitigating both gradient explosion and vanishing. Moreover, our method also achieves the best performance among all evaluated approaches.

Building on the foundation of the scaling laws, even seemingly minor modifications—such as changes to model architecture, hyperparameter configurations, initialization schemes, or training protocols—can lead to substantial shifts in performance, particularly in the context of large-scale LLMs. Techniques like GQA and MQA exemplify this phenomenon. Although our proposed changes may appear modest, they induce significant alterations in training dynamics, especially under a large training budget. Furthermore, we observe a clear improvement in convergence speedup. Extensive empirical studies and ablation analyses further validate the simplicity and effectiveness of our approach.

Therefore, our contribution goes beyond a simple combination of existing components and offers a non-trivial advancement over prior approaches.

---

Q2. Have the authors explored scaling the model beyond 7B parameters?

A2. The models we have trained so far scale up to a maximum size of 7B parameters. In fact, as described in Section 5.2 of the paper, we have already trained a 7B-parameter MoE model with 1B active parameters using 500B training tokens, and demonstrated that HybridNorm* achieves superior performance. Additionally, we have included experiments with a 7B dense model for further comparison.

New experiment in 7B dense model. The experimental setup primarily follows the 7B model configuration outlined in [2] and the number of training tokens is 150B. The first table below presents the training loss and validation perplexity (PPL), while the second table reports the downstream evaluation metrics.

The experimental results demonstrate the clear superiority of HybridNorm* over the traditional Pre-Norm approach in the 7B model. Firstly, HybridNorm* achieves a lower training loss (2.430 vs. 2.469), indicating more efficient optimization during training. This improvement in training dynamics translates into consistently better performance across a range of language modeling benchmarks. Specifically, HybridNorm* yields lower perplexity scores on all evaluated datasets, including C4, Books, Common Crawl, Wiki, and Wikitext 103. For instance, perplexity on the C4 dataset drops from 15.32 to 14.83, suggesting stronger generalization across both structured and unstructured corpora.

Moreover, HybridNorm* shows consistent improvements on all downstream tasks, covering various domains such as arithmetic reasoning, commonsense QA, and natural language inference. Notably, performance on BasicArithmetic improves significantly from 43.50% to 50.67%. Across the full suite of tasks—including ARC, PIQA, COPA, BoolQ, and WinoGrande—HybridNorm* outperforms Pre-Norm in every case, leading to an overall increase in average accuracy from 60.61% to 63.06%.

---

Please feel free to let us know if further clarification is needed.

Thank you for your thoughtful feedback and constructive comments. We appreciate that Reviewer gyxV finds our method novel, our experiments well-executed, and our theoretical derivation complete. Below, we address your concerns point by point.

---

Q1. From the overall perspective of the paper, the paper mainly focuses on performance improvement and training stability, but there is less interpretability analysis of the internal mechanism of HybridNorm. Why not you use some feature maps of different layers and experiments with heat maps to further verify the effectiveness of the method?

A1. Thank you very much for your insightful feedback. We fully agree that a more in-depth interpretability analysis could significantly enhance the understanding of HybridNorm’s internal mechanisms. In fact, we have taken initial steps in this direction in Appendix D.2 and D.3 of our paper, where we examine information flow through the perspectives of signal propagation and entropy dynamics during pre-training. The visualizations in Figure 7 and Figure 8 illustrate how HybridNorm influences layerwise entropy evolution and signal behavior across the network. We greatly appreciate your suggestion to incorporate feature map and heatmap-based analyses. We see this as a promising direction and plan to further expand our interpretability efforts along these lines in future work.

---

Q2. The method proposed in this paper is carried out from the model with 151M to 7B parameters, but the model with a larger parameter quantity has not been fully experimentally verified. Why?

A2. In fact, as described in Section 5.2 of the paper, we have already trained a 7B-parameter MoE model with 1B active parameters using 500B training tokens, and demonstrated that HybridNorm* achieves superior performance. Additionally, we have included experiments with a 7B dense model for further comparison below.

New experiment in 7B dense model. The experimental setup primarily follows the 7B model configuration outlined in [2] and the number of training tokens is 150B. The first table below presents the training loss and validation perplexity (PPL), while the second table reports the downstream evaluation metrics.

The experimental results demonstrate the clear superiority of HybridNorm* over the traditional Pre-Norm approach in the 7B model. Firstly, HybridNorm* achieves a lower training loss (2.430 vs. 2.469), indicating more efficient optimization during training. This improvement in training dynamics translates into consistently better performance across a range of language modeling benchmarks. Specifically, HybridNorm* yields lower perplexity scores on all evaluated datasets, including C4, Books, Common Crawl, Wiki, and Wikitext 103. For instance, perplexity on the C4 dataset drops from 15.32 to 14.83, suggesting stronger generalization across both structured and unstructured corpora.

Moreover, HybridNorm* shows consistent improvements on all downstream tasks, covering various domains such as arithmetic reasoning, commonsense QA, and natural language inference. Notably, performance on BasicArithmetic improves significantly from 43.50% to 50.67%. Across the full suite of tasks—including ARC, PIQA, COPA, BoolQ, and WinoGrande—HybridNorm* outperforms Pre-Norm, leading to an overall increase in average accuracy from 60.61% to 63.06%.

---

Thank you for pointing out our type. We have corrected it accordingly. Please feel free to let us know if further clarification is needed.

Thanks for your time and constructive comments. We appreciate that Reviewer RJ3j finds our method effective and consistent with our theoretical analysis. We address your concerns below and hope our responses adequately clarify your questions.

---

Q1. Could the authors elaborate on the specific differences in how RMSNorm and LayerNorm might influence the theoretical gradient analysis and the empirical performance of HybridNorm?

A1. In the following, we demonstrate that RMSNorm and LayerNorm exhibit no fundamental differences, both in theoretical analysis and empirical observations. It is worth noting that, given the vast majority of popular LLMs are based on RMSNorm, our experiments and conclusions are broadly applicable to standard LLM architectures.

On the theoretical side, we adopt the notation introduced in Appendix B of the paper. Suppose the input is given by $X \in \mathbb{R}^{s \times d}$. Let $P= I - \frac{1}{d}1_d1_d^{\top}$ . Then $\mathbb{E}X = X \frac{1}{d}1_d1_d^{\top}$ and $X-\mathbb{E}X =X - X \frac{1}{d}1_d1_d^{\top}=XP$. For simplicity, we omit the affine transformation in the normalization layers. It follows that

$$LayerNorm(X)=\frac{X-\mathbb{E}X}{Var(X)}=\frac{X-\mathbb{E}X}{RMS(X-\mathbb{E}X)}=\frac{XP}{RMS(XP)}=RMSNorm(XP).$$

Hence, analogous to the gradient of RMSNorm (Eq. (20) in the paper), the gradient of LayerNorm is given by

$$\frac{\partial LayerNorm(X)}{\partial X} = \frac{\partial RMSNorm(XP)}{\partial X} = \mathrm{blockdiag}\left(\frac{\partial RMSNorm((XP)_i)}{\partial X_i}\right)=\mathrm{blockdiag}\left( \frac{\sqrt{d}}{||(XP)_i||_2}\left(I_d - \frac{(XP)_i(XP)_i^\top}{||(XP)_i||_2^2} \right)P\right).$$

Here, $X_i$ means that the i-th column of $X$. Note that $P$ is a positive semidefinite matrix with eigenvalues bounded between 0 and 1, hence $||P||_2 = 1$. As a result, the gradient norms of LayerNorm and RMSNorm differ only by a constant factor. This implies that the main result, Theorem 2 in the paper, also holds for LayerNorm.

On the experimental side, we conducted controlled comparison using models of identical size (550M parameters), each trained on 400B tokens. Both models adopt the Pre-Norm architecture and employ the Megatron initialization scheme; the only difference lies in the normalization method—one uses RMSNorm, while the other uses LayerNorm. Detailed model configurations and experimental hyperparameters are provided in Table 6 and Table 7 of the paper. As shown in the table below, the training losses of RMSNorm and LayerNorm are nearly identical, with a marginal difference of only 0.0008. In the context of large-scale models, a loss difference smaller than 0.001 is typically considered negligible.

---

Q2. A more direct and in-depth empirical comparison with these advanced normalization methods could further solidify the superior performance of HybridNorm.

A2. We have extended Table 4 to include DeepNorm, Sandwich-LN, and OutputNorm (OLMo-2 [2]), as shown below. We have also clarified in the revised manuscript that OutputNorm refers to the normalization strategy employed in OLMo-2, where RMSNorm is applied to the outputs of attention and MLP sublayers.

It shows that HybridNorm consistently outperforms all other normalization strategies, including the aforementioned advanced methods, across all metrics. This highlights the strength of our approach in both generative perplexity tasks and HellaSwag.

---

Q3. This paper provides limited technical contribution. The method is very similar to Figure 6 in [1], merely replacing pre-norm with post-norm.. Both of the pre- and post-norm have advantages, they have their own shortcomings, too. You used post-norm, which introduces the Gradient Vanish problem.

A3. First and foremost, we would like to emphasize that our method, HybridNorm, is a simple yet effective approach to enhancing both model performance and training stability. The core idea is to integrate Pre-Norm and Post-Norm within each Transformer layer, to balance training stability and model performance. Moreover, HybridNorm* introduces a specialized treatment for a specific layer.

The primary difference between our method and the approach shown in Fig. 6 of [1] (which we refer to as QKV-Pre) lies in the integration of different normalization types within each transformer block. The layer-wise gradient norm of QKV-Pre, similar to Pre-Norm illustrated in Figure 2, tends to exhibit gradient explosion in deeper models, while Post-Norm suffers from vanishing gradients. In contrast, HybridNorm adopts a hybrid strategy within each transformer block, leading to more stable gradient propagation during backpropagation and effectively mitigating both gradient explosion and vanishing. Moreover, our method achieves the best performance. We also provides theoretical insights for QKV-Norm.

Building on the foundation of the scaling laws, even seemingly minor modifications—such as changes to model architecture, hyperparameter configurations, initialization schemes, or training protocols—can lead to substantial shifts in performance, particularly in the context of large-scale LLMs. Techniques like GQA and MQA exemplify this phenomenon. Although our proposed changes may appear modest, they induce significant alterations in training dynamics, especially under a large training budget. Furthermore, we observe a clear improvement in convergence speedup. Extensive empirical studies and ablation analyses further validate the simplicity and effectiveness of our approach.

Therefore, our contribution goes beyond a simple combination of existing components and offers a non-trivial advancement over prior approaches.

---

Q4. Please empirically validate HybridNorm on 7B or larger models.

A4. In fact, as described in Section 5.2 of the paper, we have already trained a 7B-parameter MoE model with 1B active parameters using 500B training tokens, and demonstrated that HybridNorm* achieves superior performance. Additionally, we have included experiments with a 7B dense model for further comparison.

New experiment in 7B dense model. The experimental setup primarily follows the 7B model configuration outlined in [2] and the number of training tokens is 150B. The first table below presents the training loss and validation perplexity (PPL), while the second table reports the downstream evaluation metrics.

The experimental results demonstrate the clear superiority of HybridNorm* over the traditional Pre-Norm approach in the 7B model. Firstly, HybridNorm* achieves a lower training loss (2.430 vs. 2.469), indicating more efficient optimization during training. This improvement in training dynamics translates into consistently better performance across a range of language modeling benchmarks. Specifically, HybridNorm* yields lower perplexity scores on all evaluated datasets, including C4, Books, Common Crawl, Wiki, and Wikitext 103. For instance, perplexity on the C4 dataset drops from 15.32 to 14.83, suggesting stronger generalization across both structured and unstructured corpora.

Moreover, HybridNorm* shows consistent improvements on all downstream tasks, covering various domains such as arithmetic reasoning, commonsense QA, and natural language inference. Notably, performance on BasicArithmetic improves significantly from 43.50% to 50.67%. Across the full suite of tasks—including ARC, PIQA, COPA, BoolQ, and WinoGrande—HybridNorm* outperforms Pre-Norm in every case, leading to an overall increase in average accuracy from 60.61% to 63.06%.

---

Hope our explanations address your concerns. Please feel free to let us know if further clarification is needed.

[1] Rybakov, et at. Methods of improving LLM training stability. arXiv preprint arXiv: 2410.16682

[2] OLMo Team, et al. 2OLMo 2 Furious. arXiv preprint arXiv:2501.00656