Understanding Layer Significance in LLM Alignment

We sincerely appreciate your positive and thoughtful feedback! Below, we address your concerns (W1 & W3 & Q2) and clarify any potential misunderstandings (W2 & Q1).

W1: The evaluation might have limitations.

It relys heavily on GPT-4o for conversational metrics. Human evaluations or adversarial testing could've strengthened validity.

Thank you for highlighting this limitation. We agree that relying solely on GPT-4o for conversational metrics may introduce biases and may not fully capture nuanced human judgments (e.g., coherence, engagement, social appropriateness). Incorporating human evaluations and adversarial testing would indeed offer a more robust and comprehensive assessment.

However, conducting large-scale human evaluation was infeasible due to the scope of our experiments, which involve evaluating multiple baseline models and our method across three alignment datasets and two evaluation datasets. Therefore, following prior studies [1,2], we opted to prioritize broad coverage through automated evaluation. We will explicitly acknowledge this trade-off in the Limitations section.

[1] SimPO: Simple Preference Optimization with a Reference-Free Reward

[2] Zephyr: Direct distillation of lm alignment

W2: Some claims need to be verified.

Some claims need to be verified. especially claims about broader applicability to reasoning lack thorough analysis.

We fully agree that drawing definitive conclusions requires broader evaluation across diverse architectures. The preliminary experiment on Qwen2.5-7B-Instruct (Sec. 5, Fig. 3, Tab. 10) serves as evidence of the Potential (stated in line 71) of our method to enhance reasoning capabilities. This experiment was conducted shortly after the reasoning datasets became available, around February and March.

W3: Reproducibility.

Details on hyperparameter tuning (e.g., LoRA ranks, learning rates) are referred in appendix, but may not reproducible

Below, we clarify our experimental setup and how we ensure fair comparisons and hyperparameter reproducibility:

1. LoRA Rank (Fixed for Fairness): We set the LoRA rank to 32 for all methods to ensure consistency and avoid any confounding effects from hyperparameter tuning. Although our initial results indicate that higher ranks (such as 64) might improve performance, we chose to keep the rank fixed. This allows us to focus on evaluating the impact of our method, rather than differences caused by varying hyperparameters.
2. Learning Rate (Tuned via Two-Stage Search): For each method, we first fixed a reasonable learning rate and tuned the number of training epochs. After identifying the optimal number of epochs, we then searched for the best learning rate using this fixed epoch count. This avoids an exhaustive grid search while still enabling effective optimization.
3. LLM Evaluation Variance: We acknowledge that evaluations involving GPT-4o are inherently sensitive to API versions and configuration settings (e.g., prompts, decoding parameters). Despite this, all models—including our baselines—were evaluated under identical conditions to ensure fairness.
4. Code Release for Reproducibility: To ensure reproducibility, we will publicly release our code and configuration files upon publication.

Q1 (Part 1): How does the binary mask balance sparsity and performance during gradient descent?

We believe there may be a misunderstanding regarding our training process. Our method does not require balancing sparsity and performance during gradient descent. Instead, for each layer, we optimize a real-valued variable $\\{s_t^i\\}_{i=1}^N$, which is mapped to the (0, 1) range using a sigmoid function $\sigma(s_t^i)$. This score reflect the importance of each layer. After learning these scores, we use a hyperparameter $K$ to select the top-$K$ layers based on their scores for subsequent selective fine-tuning. Therefore, the sparsity of the binary mask is determined solely by the hyperparameter $K$.

Q1 (Part 2): Could stochasticity in training affect layer importance rankings?

We already addressed this from two perspectives in our paper:

1. Random seeds: As shown in Table 3, the layer importance rankings are highly consistent across different random seeds, with a Jaccard Similarity of nearly 90%, indicating strong robustness to training stochasticity.
2. Parameter initialization: As discussed in our ablation study (Observation 5, Table 15, Appendix C.4), different initializations lead to only minor variations, while the high Jaccard Similarity indicates that the rankings remain generally stable.

We will emphasize these points more clearly in the revised version.

Q2: How would ILA perform on alignment datasets with conflicting stylistic objectives (e.g., formal vs. casual responses)?

Thank you for your thoughtful question. To address this, we conducted additional experiments on the OASST1 dataset [1], which features diverse linguistic styles and perspectives, contributed by volunteers with varying educational and professional backgrounds. This diversity makes OASST1 well-suited for analyzing stylistic variation. Simlar to QLoRA [2], we use a subset of OASST1 dataset, which contains the highest-rated paths in the conversation tree.

• Robustness of Layer Importance Rankings: We compared those obtained by fine-tuning LLaMA 2-7B on OASST1 with rankings from LIMA, No Robots, and Alpaca-GPT4. As shown in Table 1, the Jaccard Similarities remain high (0.83–0.90), indicating that ILA identifies similar important layers despite stylistic variation.

Table 1: Jaccard similarities of the top 75% important layers in Llama 2-7B.

Note that these similarities are slightly lower than those observed among LIMA, No Robots, and Alpaca-GPT4 themselves (∼91%; see Table 2 in the main paper), which is expected given the broader stylistic and linguistic diversity present in OASST1.

• Performance Improvements: We also fine-tuned LLaMA 2-7B on OASST1 using only the top ~75% important layers identified by ILA. As shown below, selective fine-tuning yields consistent performance gains:

Table 2: Comparison of Llama 2-7B fine-tuned on the OASST1 datasets.

These findings demonstrate that ILA remains effective even on datasets with conflicting stylistic objectives.

[1]Köpf, Andreas, et al. "Openassistant conversations-democratizing large language model alignment. CoRR, abs/2304.07327, 2023. doi: 10.48550." arXiv preprint arXiv.2304.07327.

[2]Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." Advances in neural information processing systems 36 (2023): 10088-10115.

We sincerely appreciate your thoughtful and constructive feedback! Below, we address your concerns.

W1: Reliance on LoRA Approximation.

The ILA method primarily utilizes LoRA (Low-Rank Adaptation) to approximate parameter changes for computational efficiency. An ablation study indicates that LoRA-based importance scores achieve nearly 83% overlap with those derived from full fine-tuning (FFT). However, this also implies that LoRA is an approximation and may not perfectly mirror the true parameter changes of FFT, which could influence the precise identification of the most critical layers. Despite this, the paper suggests LoRA provides a "strong approximation".

Why LoRA is considered a strong approximation of FFT:

1. Our experiments show that LoRA’s importance rankings overlap with those from FFT by 83%. This high level of agreement is sufficient for selective tuning, which is inherently robust to small deviations in layer ranking. Minor discrepancies (e.g., the 17% non-overlap) have minimal impact on downstream performance, enabling substantial reductions in computation while maintaining accuracy.
2. Furthermore, we identified layers marked as important (Top 75%) by FFT but not by LoRA on LLaMA 2-7B using the No Robots dataset. These layers had low average ranks according to FFT (124 out of 168; best rank: 57), suggesting they are not among the most critical layers.

While we chose LoRA for its efficiency, FFT remains a practical alternative. Because our method is robust across datasets, using FFT to identify layer importance rankings for a specific model incurs only a one-time cost. Users can then directly reuse the precomputed FFT-based rankings for different alignment tasks and similar model architectures.

W2: Computational Cost of Initial Stage.

Although ILA is presented as an efficient method, its first stage, which involves training the model with LoRA until it reaches an stable state, accounts for the majority of the computational cost. The paper suggests that identifying layer importance effectively requires only 25-50% of the training milestones. Nevertheless, this initial phase can still be resource-intensive, particularly for very large models.

The key insight of this work is that layer importance rankings are consistent across different alignment datasets (Sec. 5.2), which enables the use of a single ranking across multiple tasks. This consistency allows us to provide standardized importance rankings for popular models (e.g., LLaMA-3, Qwen2.5), eliminating the need for repeated computation. Consequently, the overhead of identifying important layers with ILA is a one-time cost.

W3: Exploration Range for Layer Freezing/Tuning.

The study primarily investigates freezing approximately 25% of unimportant layers to enhance performance or fine-tuning 10-30% of key layers to improve efficiency. While these specific percentages yield positive outcomes, a more detailed analysis of how varying these proportions affects the balance between performance and efficiency could offer further insights.

Our analysis already demonstrated the trade-off between performance and efficiency. Specifically, fine-tuning ~75% of layers (Tables 5, 17, and 18) achieves strong performance, while tuning only ~30% of layers (Table 6) retains reasonable performance with improved efficiency.

To further investigate this, we conducted additional experiments by fine-tuning Llama 3.1-8B on the No Robots dataset with varying proportions of layers (15%, 35%, 55%, 75%). As summarized in the Table below, the results reveal a non-linear trade-off: training efficiency steadily increases as fewer layers are tuned, while performance remains relatively stable when tuning 75% or 55% of the layers, but drops considerably when tuning only 35% or 15% of the layers.

Table 1: Impact of the Proportion of Fine-tuned Layers on Performance and Efficiency (Model: Llama 3.1-8B; Alignment Dataset: No Robots). Batch size is 1 and maximum token length is 1024. Training time denotes the average time per iteration.

Dear Reviewer 1beu

We submitted our rebuttal four days ago, and as the discussion period is now nearly halfway over, we would like to kindly inquire whether our responses have sufficiently addressed your concerns. If there are any remaining questions or issues that require clarification, we would greatly appreciate your feedback.

We sincerely hope that our responses meet your expectations and may contribute to a reconsideration of your evaluation. Thank you once again for your time and consideration.

Best regards,
The Authors

We sincerely appreciate your positive feedback and thoughtful suggestions!

However, I do feel the paper can be further improved by trying to answer "why" those layers are important, which will make the paper more novel.

We fully agree with your thoughtful suggestion. Our layer ranking results (e.g., Figure 1) consistently demonstrate that Feed-Forward Network (FFN) layers are more important than attention layers. This observation aligns with previous studies [1,2], which identify FFNs as key memory modules responsible for encoding factual knowledge, while attention layers primarily facilitate token interactions. This distinction explains why FFN layers consistently achieve higher importance rankings in our results.

Moreover, our findings suggest several practical implications:

• Model compression should prioritize retaining upper FFN layers.
• Knowledge editing may be more effective when targeting middle-to-upper FFN layers.

We will revise Section 4.2 to emphasize this observation and its implications, which we believe will further strengthen the novelty of our contribution.

Thank you once again for your invaluable suggestion!

[1] Geva, Mor, et al. "Transformer feed-forward layers are key-value memories." arXiv preprint arXiv:2012.14913 (2020).

[2] Meng, Kevin, et al. "Locating and editing factual associations in gpt." Advances in neural information processing systems 35 (2022): 17359-17372.

Q1: Try use vector graphics in Figure 2.

Thank for your valuable advice. We will use vector graphics in the revised revision.

We sincerely appreciate your positive comments and thoughtful feedback! Below, we address some potential misunderstandings (W1 & W3) as well as your additional concern (W2).

W1: overclaim the potential of the proposed method.

Selective fine-tuning requires the identificaiton of the important layers first using ILA, which involves at least 30-50% of normal training (either full-model training or PEFT training) and the score training overheads. I do not think selective fine-tuning helps much with improving training efficiency, as gradients still need to be propagated to earlier layers.

Please note that identifying important layers via ILA incurs only a one-time cost (30–50% of standard training). As shown in Section 5.2, layer importance rankings are consistent across different alignment datasets, allowing us to reuse a single ranking across tasks. This enables us to provide standardized importance rankings for popular models (e.g., LLaMA-3, Qwen2.5), eliminating the need for repeated computation.

Regarding training efficiency, our method primarily improves memory efficiency, while also contributing to overall training speed. As shown in Table 11 (Appendix C.1), our method reduces peak GPU memory usage by approximately 59% in full fine-tuning and offers an additional ~30% reduction on top of efficient methods like QLoRA. While the speed improvement is less pronounced, we still observe an increase of over 10% in training speed.

We will clarify these points further in the revised version. Thank you again for your valuable feedback.

W2: The absence of AdaLoRA+ILA baseline.

I am not sure why the authors include AdaLoRA as a baseline but do not include results of AdasLoRA with ILA. If I am understanding it correctly, AdaLoRA is a LoRA variant and is orthogonal to ILA.

Thanks for the thoughtful question.

We chose AdaLoRA as a baseline because it shares a similar objective with ILA. Both AdaLoRA and ILA aim to allocate tuning resources based on layer importance. AdaLoRA achieves this implicitly by adjusting ranks during training (i.e., rank=0 in AdaLoRA indicates a layer does not require tuning), while our ILA explicitly identifies the important layers to be tuned prior to the fine-tuning process.

Indeed, AdaLoRA could be combined with ILA. However, the current implementation of AdaLoRA in the PEFT library contains bugs that significantly hinder distributed training, leading to prohibitively slow execution. As a result, we were unable to conduct this experiment in our submission due to limited computing resources.

During the past few days of the rebuttal period, we have experimented with combining AdaLoRA with ILA on LLaMA2-7B, using the No Robots dataset for alignment. The test results in the Table below show that incorporating ILA consistently improves performance.

Table 1: Comparison of Llama 2-7B fine-tuned on the No Robots dataset.

W3: The message conveyed by Thm 3.1 is unclear.

Firstly, the first statement in Thm 3.1, "For a sufficiently small $\epsilon$, $\theta_T$, is $\epsilon$-stable" looks an assumption to me. The statement itself reads wierd to me because we do not know if a pre-trained transformer is $\epsilon$-stable for a very small $\epsilon$. We can only say that it is $\epsilon$-stable for a sufficiently large $\epsilon$. But a large $\epsilon$ makes the result in eqn (11) weak. In summary, I am confused about Thm 3.1.

Please note that the $\epsilon$-stability assumption in Theorem 3.1 does not refer to the model being $\epsilon$-stability during the pre-training phase. Instead, this assumption applies specifically to the alignment training stage. At this stage, after the loss has largely converged, we assume $\theta_T$ is $\epsilon$-stable, which implies that the model parameters are close to a local optimum and change very little between iterations.

This assumption is supported by standard optimization theory (e.g., gradient descent on smooth non-convex objectives [1]), which shows that as training progresses, the difference in expected loss between consecutive steps approaches zero: $\\lim_{t \to \\infty} \\left|E_{z}[\mathcal{L}(\\theta_{t+1})] - E_{z}[\mathcal{L}(\\theta_t)] \\right| \to 0$. This justifies assuming a small $\epsilon$ at convergence.

As stated in line 148-149, Theorem 3.1 shows that when $\theta_T$ is $\epsilon$-stable, solving the optimization problem in Eq. (3) for any $t > T$ will yield similar results —specifically, the similar binary masks will be learned.

We'll further clarify these points in the revised version.

[1] Optimization methods for large-scale machine learning. SIAM review 60.2 (2018): 223-311.