Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models

We thank the reviewer for their valuable feedback and for taking the time to review our work. Please find our responses to each of your points below.

## 1. Experimental Scope and Design Choices

"Experiments are not thorough. For example, only two types of modifications, i.e., expansion and removal, have been proposed. Why not narrow down but removal? Why do not leave the original width at all?"

We appreciate this insightful suggestion about exploring FFN width reduction rather than complete removal. This is indeed an interesting research direction that we did not explore due to the limited rebuttal period. If this paper is accepted, we will commit to investigating these additional configurations (including narrowed FFNs and retaining the original width) for the camera-ready version.) We acknowledge that the space of possible experimental configurations is vast, and we cannot exhaustively explore all variations within our resource constraints. However, within our defined scope, systematic exploration of FFN expansion and removal across multiple scales, positions, and ratios, our work provides meaningful contributions to understanding the layer-wise FFN importance. Our findings establish a foundation for future investigations, including the promising directions suggested by the reviewer.

## 2. The Size of Experimental Models

“The sizes of experimented models (with 285M to 1.2B parameters) are relatively small, … 80-layer and 126-layer models investigated under the name of Llama (70B and 405B models).”

Our comprehensive study required substantial resources: 5,500+ GPU hours (H100) across 54 model configurations (285M-1.2B parameters). We followed a systematic approach: exhaustive small-scale exploration followed by targeted validation at larger scales. The consistent patterns across our 3× parameter range (285M → 1.2B) provide strong evidence for transferability.

COLM's guidelines recognize that "most researchers do not have access to large-scale compute" and that limiting research to well-resourced labs "will stifle innovation." Our work exemplifies meaningful architectural insights within academic constraints. Training larger models during the rebuttal is practically impossible, requiring weeks of computation beyond available timelines.

Our systematic exploration reveals novel layer-wise FFN importance patterns that provide specific hypotheses for future large-scale validation, rather than requiring complete re-exploration at larger scales.

## Answers to the Questions To Authors:

First of all, the paper does not explicitly state whether they are investigating encoders or decoders. Readers can know they are decoder-only models only when they see "LLaMA" in line 65.

Thank you for pointing this out. As you stated, we mainly focus on decoder-only models. We thought that the main focus in the LM community recently has been that Transformer-based LMs are decoder-only models. We will refine our wording to clarify that our focus is on decoder-only models for this paper in the camera-ready version if our paper is accepted to the conference.

Propose method modifies several layers. Then, can the resulting models still be called "LLaMA architecture" (ll.70-71) ?

Thank you for pointing this out as well. However, we would like to emphasize that we have already distinguished the LLaMA architecture from our modified models, which we refer to as experimental models. In other words, we did not intend to refer to our modified models as the “LLaMA architecture.”

Please confirm that the reference in lines 70–71 only describes the baseline and the starting point of the experimental models. Subsequently, we consistently distinguish between baseline and experimental models using the “experimental model” or “[position][ratio][size]_[layers]” format throughout the paper.

In any case, we will certainly refine our wording to avoid such confusion in the next version.

How the authors come up with this "novel" approach (l.3)? In other words, the rationale should be explained more precisely.

Our approach is based on the observation from existing research that FFNs serve as a knowledge storage mechanism in Transformer LMs. This observation naturally raised questions for us: “Does the importance of FFNs depend on the layer position?” and “Can Transformer LMs properly train knowledge even by eliminating FFNs from certain layers?”

From these questions, we aimed to empirically investigate the relationship between performance and the importance of FFN positions in Transformer LMs with unevenly located FFNs. Note that we attempted to directly verify FFN positional importance through architectural modification and training from scratch, unlike many previous studies that analyze pre-trained models without architecture modification.

The expanded width, i.e., df′, in the r% layers is determined on the basis of the number of (100-r)% of FFN-removal layers. Please precisely describe that equation. I'm wondering if the variants models can have exactly the same number of parameters.

To maintain equal parameter counts, d'_f is calculated as: d'_f = d_f/r + (1-r)/(3r) ≈ d_f/r (when d_f is sufficiently large). We determine the intermediate dimension using the nearest integer value. This will be clarified in the camera-ready version if our paper is accepted to the conference.

Accuracy and perplexity of the baseline models, ... making it unclear if the RI is worth discussing and its scale.

As noted in the footnote on page 5, tasks with performance below the chance level are excluded, ensuring the validity of the RI score. The baseline performance is available at: https://anonymous.4open.science/r/layer_wise_importance_of_ffn_for_rebuttal-3E06. We will include this in the camera-ready version.

Some numbers in Section 5 are not corresponding to those in Figure 4... The IDs in l.280 cover 9, which does not match 10 in l.283. Similarly, mentions about layers in ll.298-302 often do not match with those in Figure 5.

Thank you for identifying these inconsistencies. We will thoroughly verify all numerical correspondences between figures and text to ensure consistency in the camera-ready version.

Although it is commonly done, I'm not still sure if one can take "average" of different metric scores with different scales, as in Sections 5.3 and 5.4. Could you give a justification?

Our justification includes: (1) Using relative improvement (RI) scores normalizes different scales, (2) Following established academic practices in language model research[1, 2, 3, 4], and (3) Confirming result robustness through consistent trends across individual tasks and averages.

[1] Grattafiori et al. 2024, The Llama 3 Herd of Models

[2] Raffel et al. 2019, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

[3] Chowdhery et al. 2022, PaLM: Scaling Language Modeling with Pathways

[4] Le Scao et al. 2022, BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

The superiority of "middle_70" for Wikitext, LAMBADA, HellaSwag, and zsRE ...the largest performance drop in the new downstream tasks, i.e., ARC-c and Winogrande?

While training convergence is confirmed (see the provided link above), computational constraints limit the model size and training tokens. ARC-c and Winogrande require complex reasoning**, which** may be challenging under our limited training setup.

## 4. The Size of Experimental Models

"Small-scale experiments: I understand that access to computing resources may be limited, but 1.2B parameters is still considered a very small model in today's terms."

Our comprehensive study required substantial resources: 5,500+ GPU hours (H100) across 54 model configurations (285M-1.2B parameters). We followed a systematic approach: exhaustive small-scale exploration followed by targeted validation at larger scales. The consistent patterns across our 3× parameter range (285M → 1.2B) provide strong evidence for transferability.

COLM's guidelines recognize that "most researchers do not have access to large-scale compute" and that limiting research to well-resourced labs "will stifle innovation." Our work exemplifies meaningful architectural insights within academic constraints. Training larger models during the rebuttal is practically impossible, requiring weeks of computation beyond available timelines.

Our systematic exploration reveals novel layer-wise FFN importance patterns that provide specific hypotheses for future large-scale validation, rather than requiring complete re-exploration at larger scales.

We thank the reviewer for their valuable feedback and for taking the time to review our work. Please find our responses to each of your points below.

## 1. Standard Deviation and Statistical Significance

We understand the importance of conducting multiple training runs with different seeds when evaluating different architectures, but this is not practically feasible due to computational resource constraints. As a general approach, numerous strong papers have adopted the method of conducting single training runs with fixed seeds and demonstrating model superiority by evaluating on more tasks [1, 2, 3, 4]. The downstream task evaluations in these papers use greedy sampling strategies and do not output standard deviations from multiple trials. Therefore, we are evaluating our method in an academically justified setting.

[1] Grattafiori et al. 2024, The Llama 3 Herd of Models

[2] Raffel et al. 2019, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

[3] Chowdhery et al. 2022, PaLM: Scaling Language Modeling with Pathways

[4] Le Scao et al. 2022, BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

## 2. Insufficient Diversity in Base Architecture

"Only one architecture was used throughout the study... there is currently a clear trend toward mixture-of-experts (MoE) architectures"

We understand the reviewer's observation regarding the development of MoE architectures. However, we respectfully maintain that LLaMA represents the most appropriate choice for our research objectives, though we acknowledge the need to clarify our "de facto standard" characterization. Rationale for LLaMA selection: Our study aims to isolate and understand the fundamental mechanisms of FFN layer importance. This requires a controlled experimental environment where we can attribute performance changes specifically to FFN positional modifications rather than confounding architectural factors. LLaMA provides this foundation through: Integration of modern improvements (Pre-LN, SwiGLU, RoPE) ensuring relevance to contemporary models Extensive academic validation and reproducible implementations Well-understood component behaviors that allow clean isolation of FFN effects Comparison with alternative architectures: While we appreciate the suggestion to explore GPT-2 (LayerNorm + GELU), its lack of modern architectural improvements would compromise the relevance of our findings to current model development. Similarly, MoE architectures, despite their growing importance, introduce additional complexity through expert selection mechanisms and sparse activation patterns that would confound our ability to measure pure FFN positional effects.

## 3. Insufficient Exploration of FFN Expansion Methods

"The FFN-expansion method could have been further investigated. For example, regarding whether to add depth instead of width to the network."

Our research scope is specifically analytical, aiming to verify FFN importance within existing architectures. As stated in lines 37-38, our research question (RQ) focuses on understanding "the roles and functions of FFNs within existing Transformer-LMs, particularly how their importance varies by layer position." While adding depth is an important perspective for architectural design, it addresses a fundamentally different problem from our RQ for the following reasons: Baseline comparison validity: Adding depth fundamentally alters the architecture itself, changing our analytical target of "FFN importance within existing Transformer-LMs." Causal confounding: Depth addition conflates "positional importance changes" with "structural performance changes," preventing pure isolation of the layer position effects we aim to understand. In summary, depth expansion falls outside our research scope as it pertains to architectural design rather than our analytical investigation of FFN mechanisms within established Transformer structures.

We thank the reviewer for their valuable feedback and for taking the time to review our work. Please find our responses to each of your points below.

## 1.

"The takeways are not clearly mentioned, and the writing is a bit weak."

We respectfully disagree with the observation that the takeaways (our main contributions) are unclear. Our primary finding, namely, concentrating FFNs in 70% of consecutive intermediate layers while maintaining the same parameter count consistently outperforms standard configurations across multiple model sizes and tasks, is clearly articulated throughout the paper in the following locations: Abstract (lines 10-14), Section 5.2 (lines 233-235), Section 5.3 (lines 246-248), Section 5.4 (lines 262-267), and Conclusion (lines 310-315).

## 2.

"Since the authors are training the model from scratch can they show the impact of the best configuration (i.e. middle layer expansion) on pre-train val loss/ train loss. How the convergence and generalization looks like in these scenarios."

The following link shows the training and validation learning curves: https://anonymous.4open.science/r/layer_wise_importance_of_ffn_for_rebuttal-3E06. We will provide all the logs for training and evaluation from our experiments once this paper is accepted to the conference.

## 3.

"The results in Fig 4 are not conclusive as the gains are highly dependent on the dataset."

We acknowledge that the gains depend on the task, but this variation is unavoidable for the following reasons:

• Dependence on Task Difficulty : The extent of improvement naturally hinges on the complexity of each task—easier tasks tend to show smaller performance boosts. Since it's fundamentally impossible to objectively normalize task difficulty across different benchmarks, it's unrealistic to expect consistent gains for all tasks.
• Deliberate Evaluation Design : We've tackled this issue through meticulous experimental planning:
  • Section 4.3 details evaluations over a broad range of task types, including language modeling, commonsense reasoning, and knowledge evaluation.
  • As mentioned on page 5, we excluded tasks where the baseline model performed worse than chance.
  • We also verified consistent trends across different model sizes (285M, 570M, 1.2B) and layer counts (12, 24, 40).
• Reliable Patterns : Even though gains differ by task, we observed stable patterns across tasks and model configurations in Sections 5.1 and 5.2:
  • Low FFN expansion ratios (10–30%) consistently underperform.
  • High ratios (70–90%) often surpass the baseline.
  • Middle placement of FFN layers tends to outperform initial and final placements. These consistently observed trends across varied conditions strongly support our conclusions. Therefore, while Fig. 4 reflects task-related variability, it doesn’t weaken the credibility of our findings.

## 4.

"Not much intuition is discussed through out the paper."

An intuitive explanation of why the “Middle-70” configuration is effective:

• Theoretical background : Previous studies have demonstrated that the most significant information processing for downstream tasks primarily occurs in the mid-to-final FFN layers of a model [1,2].

• Effectiveness of Middle-70 : By concentrating FFN layers in the parts of the model that matter most for downstream tasks, the parameter budget is used more efficiently.

• Novelty : Unlike prior work that analyzed publicly released pre-trained models (static models), we trained models from scratch with different FFN settings in specific layers, yet still achieved performance equal to or better than the baseline. This reveals an intriguing aspect of the Transformer’s redundancy and adaptability.

• Real-world implications : This approach offers a promising new direction for designing efficient models that deliver higher performance within the same parameter constraints.

• [1] Meng et al. 2022, Locating and Editing Factual Associations in GPT

• [2] Geva et al. 2021, Transformer Feed-Forward Layers Are Key-Value Memories

## 5.

"Why in case of smaller models 285M (Table 1) first layer expansion outperforms the middle layer expansion?"

I would like to clarify a misunderstanding. It is not that the first layer expansion outperforms other configurations; rather, “middle 70” ranked third, surpassing the baseline by 2.14%, which supports our claim. Furthermore, in both the 570M (24-layer and 40-layer) models, “middle 70” achieved first place. Its superiority becomes even more apparent as the number of layers increases. This is likely because, as the layer count rises, the overlap between layers designated for FFN expansion in the first, middle, and final settings diminishes, causing the performance differences to appear more sharply.

We appreciate your thoughtful comments and believe that our responses clarify the contributions of our work while also outlining promising directions for future research.

We thank the reviewer for their valuable feedback and for taking the time to review our work. Please find our responses to each of your points below.

##1. Contribution to Practicality and Architecture Search

The down side of the work is that it focus much on a niche topic; and it might be hard to connect it to a boarder use case like "how would the message in the paper help people do neural architecture search?" or "what's the next neural architecture?".

This research addresses an important and practical topic for the following reasons: FFNs are a core component because they occupy approximately 2/3 of the parameters in transformers, and their optimization directly impacts overall model performance. From this perspective, we believe that the research on FFN placement strategies is not niche but rather one of the most critical research topics for improving transformers.

Practical value of this research: fundamental questioning of existing paradigms – we empirically demonstrated that the currently mainstream uniform placement across all layers may not be optimal.

New direction for architecture search: By showing that a middle 70% placement consistently achieves superior performance across multiple model sizes, we have opened up a new dimension of Neural Architecture Search involving non-uniform FFN parameter strategies.

##2. Assumption of Equal Importance for FFN-Expanded Layers

The authors assumes FFN-expanded are equally important. However, this likely might not be the case as reflected in Figure 5.

The "equal importance" assumption you pointed out is not a claim that FFN-expanded layers are actually equally important, but rather a methodological starting point we adopted because directly quantifying the individual contribution of each layer in specific configurations is technically extremely challenging. Interestingly, despite this simplified approach, we observed a clear trend of consistently higher importance for middle layers across multiple model configurations, suggesting that our analytical method can significantly capture actual layer-wise importance differences. Your suggestion of non-uniform allocation based on importance to the "blue" layers in Figure 5 is a natural and important next step for this research. Allocating more parameters to layers with higher importance scores could be expected to yield further performance improvements, making this a valuable research direction to pursue.

##3. Relationship with Research Using Influence Functions

Have the author used techniques from [1], would the layer importance conclusion differ? [1] Studying Large Language Model Generalization with Influence Functions

The paper [1] you mentioned employs a post-hoc analytical approach that examines the layer-wise influence of training data on already pre-trained models with standard Transformer architectures.

In contrast, our research takes a fundamentally different approach by modifying the model architecture itself from the beginning of training, such as removing FFNs from certain layers during the pre-training stage. Since [1] analyzes standard architectures after training completion while our work involves architectural modifications during training, the methodologies are incompatible and orthogonal.

##4. The Method for Determining Intermediate Dimensions of FFNs in Experimental Models

Figure 3, how could the percentage be 90%?

As you mentioned, d'_f is indeed recalculated based on the proportion of FFN-expanded layers. Specifically, when 90% of layers are FFN-expanded and 10% are FFN-deactivated (completely removed), we recalculate the intermediate dimension d'_f of FFN-expanded layers according to the ratio r to maintain the same total parameter count. This mechanism ensures that at 100% ratio, all layers have standard FFNs, making the model identical to the baseline.

##5. The Axis Labels in Figure 5

Figure 5 axis label are not visible.

Thank you for your feedback. In Figure 5, the horizontal axis represents layer indices, and the vertical axis represents the layer-wise importance scores (standardized) calculated according to Appendix D. We will improve the visibility issues in the camera-ready version.