Information Guided Regularization for Fine-tuning Language Models

Thank you for taking the time to review our manuscript. Your insightful comments help us better our research. Please let us address your concerns below:

1. We agree and will include this block in the revised manuscript to clarify the implementation of FIM as well as the P_upper & P_lower based schedules - see algorithm.

2. While we agree that these papers related to FIM & NN training would provide a fruitful background to our readers, they are solving fundamentally different problems:

Their Problem: Using parameter-efficient fine-tuning (PEFT) via sparser models, can we match standard fine-tuning using the full dataset, by computing FIM for each target task?

Our Problem: With the architecture unchanged, can we beat standard fine-tuning, especially in scenarios of data paucity, by computing FIM only once for the source task?

Nevertheless, we did the comparison suggested by the reviewer and our method indeed beats the latest state-of-the-art in the list (Lodha 2023) by a significantly large margin, see table.

Regarding the other baselines mentioned:

• Liu 2024 use FIM as an analytical tool for their chosen domain of cross-lingual adaptation rather than as a general purpose regularizer, as done here. Further, this paper was published on Arxiv after the COLM submission deadline.

• LP-FT, gradual unfreezing (similar to Liu 2024), and auto-RGN/SNR are PEFT as well. RGN/SNR (similar to Sung 2021 & Lodha 2023) are based on target tasks.

To address your questions:

1. With a grid-search, we found P_lower = 0% and P_upper = 10% with a linear increment schedule to be the most effective approach. We will add grid-search results to the Appendix if our manuscript is accepted. Thank you for bringing this to our attention.

2. Consider A to be the FIM with scores for each parameter computed w.r.t the full corpus. Table 1 evaluates A with matrices B_n - FIMs with scores for each parameter computed w.r.t n % of the corpus. From observation 3, we know that the information from FIMs we need to pull only pertains to the top x % of parameters. Thus, The FIM approximates B_n are not computed w.r.t a subset of parameters but are rather evaluated w.r.t them.

3. Yes. The central requirement for guided dropout to function is the fact that modern LMs are overparameterized. In section 2.3, we show that this thesis holds true for both GPT2 (decoder based) and T5 (encoder-decoder based) models.

Thank you for taking the time to review our manuscript and for your positive remarks. These insightful comments help us better our research. Please allow us to address your concerns and the questions you have raised:

1. We agree that extending our findings to a greater array of models and datasets would further help our cause. With statistics presented for 5 random restarts, showcasing the efficacy of our approach under data sparsity for downstream tasks requires fine-tuning all models with several cuts of the training dataset (from 100% to 10%). Thus, to accommodate both compute and page constraints, we chose the tried and tested GLUE benchmark. However, if our manuscript is accepted, we will utilize the extra page to extend our experimentation to the SuperGLUE benchmark [1] as well as other models.

2. Yes, the effectiveness of our approach indeed relies on the accurate estimation of the Fisher information. Thus, through observation 2 & table 1, we establish that for a text-based pretraining corpus (central to all LLMs), the Fisher information can be reliably estimated using as little as 1% of the training data (rows 4 & 8 in table 1), making our method tractable even for large complex datasets and models.

3. The foundational principle behind our approach is the fact that modern LMs are overparameterized. This overparameterization leads to a small subset of model parameters driving the training loss geometry (observation 3, page 4). Following this, our claim for architectural agnosticism mainly stems from two observ​​ions:

• In section 2.3, we show that the same overparameterization that held true for the encoder-only BERT also holds true for the decoder-only GPT2 and the encoder-decoder based T5. Although modern LMs such as GPT3/4, Llama, and Mistral are significantly larger, they operate on the same decoder-only architecture as GPT2.

• Recent work has shown that these larger decoder-only models are as overparameterized (if not more) than their smaller counterparts. Gromov et. al. 2024 [2] showcase that up to 50% of Llama 70B, 40% of Llama 13B, and 30% of Mistral parameters are redundant. Thus, although we personally do not have the compute to evaluate larger models, we strongly believe our findings should generalize.

References:

1. Wang et. al. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. 2020.

2. Gromov et. al. The Unreasonable Ineffectiveness of the Deeper Layers. 2024.

Thank you for taking the time to review our manuscript and for your positive remarks. These insightful comments help us better our research. Please allow us to address your concerns and the questions you have raised:

1. If our manuscript is accepted, we will append the following details to section 2:

• For a network with |W| parameters, the cost of estimating Fisher information using n samples is O(n|W|) gradient computations in the network. We showcase that reliable estimates of Fisher information can be obtained with n as tiny as 1% of the pretraining corpus (table 1).

• While our current plots showcase sorted layer-wise Fisher scores, we will append figures (see here) to showcase the raw layer-wise distributions. Interestingly, we note that vital layers are spread throughout the network and not concentrated at the beginning or end as traditionally believed.

• We will expand on the details of the loss-landscape computation and visualisation through more elaborate writing and by mathematically extending aspects of equation 5.

2. Figures 2 a & b serve to show the profound effect that a few vital parameters have on the model's loss geometry. 2(b) shows that perturbing the top 0.25% of parameters has the same effect on the loss geometry as perturbing the top 40%. You raise an important question - where is the tipping point (40% < x < 60%) such that the degradation of the loss landscape turns from gradual to sudden. We believe this warrants an "edge of chaos" style analysis [3] which we aim to do in future work.

3. OOD generalization across similar tasks as a function of Fisher-based sharper/flatter parameters is an interesting question. There is evidence indicating that Fisher-guided pretraining-finetuning helps generalize better to OOD instances [1]. In line with this, we will evaluate OOD generalization for our method in future work.

4. We agree that if applied to modern architectures that are even more overparameterized [2], the effects of guided-dropout would be even more pronounced. Although we personally do not have the compute to evaluate larger models, we strongly believe our findings should generalize.

References:

1. Sharma et. al. Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency. ACL Proceedings. 2023.

2. Gromov et. al. The Unreasonable Ineffectiveness of the Deeper Layers. 2024.

3. Zhang et. al. Edge of Chaos as a Guiding Principle for Modern Neural Network Training. 2021.

Thank you for taking the time to review our manuscript and for your positive remarks. These insightful comments help us better our research. Please allow us to address your concerns and the questions you have raised:

1. Models like BERT/GPT2/T5, while superseded by larger models, are still useful to demonstrate the validity of theoretical constructs like ours (see other recent papers like [1,2]). Current limitations in resources prevent us from applying our approach to models like Llama/Mistral. However, as we believe all modern LMs to be overparameterized, our methods should transfer over:

Architectural Agnosticism: Overparameterization drives redundancies in the training loss geometry that can be harnessed by intelligent dropout mechanisms like ours. In section 2.3, we show that the same overparameterization that held true for the encoder-only BERT also holds true for the decoder-only GPT2 and the encoder-decoder based T5. Although modern LMs such as GPT3/4, Llama, and Mistral are significantly larger, they operate on the same decoder-only architecture as GPT2.

Prevalence of Overparameterization: Further, recent work has shown that these larger decoder-only models are as overparameterized (if not more) than their smaller counterparts. Gromov et. al. 2024 [3] showcase that up to 50% of Llama 70B, 40% of Llama 13B, and 30% of Mistral parameters are redundant. Thus, although we currently do not have the compute required to evaluate larger models, we strongly believe our findings should generalize.

2. While SSMs like Mamba [4] have showcased transformer-esque performance with faster inference and better scaling, their application to the language modality still follows the pretrain-finetune paradigm. Thus, our regularizer should improve downstream performance. Although gauging the magnitude of improvement requires a Fisher-based investigation similar to section 2. If our manuscript is accepted, we will include an investigation about the applicability of our method to non-transformer based architectures.

References:

1. Sharma et. al. Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency. ACL Proceedings. 2023.

2. Lodha et. al. On Surgical Fine-tuning for Language Encoders. EMNLP Findings. 2023.

3. Gromov et. al. The Unreasonable Ineffectiveness of the Deeper Layers. 2024.

4. Gu et. al. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023.