LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection

Thank you for your insightful questions. The followings are our answers to your concerns.

Q1: "The analysis of the paper heavily depends on Theorem 1, which is a Generalization bound of LLM’s finetuning. I am not quite sure whether the bound is tight enough to claim that model A is better than model B since its generalization bound is smaller."

A1: We would like to clarify that Theorem 1 is not used for direct model comparison. Instead, its primary purpose is to fit the transition phases and identify the transition point in the fine-tuning dynamics. Once this transition point is determined, we apply our regression model to predict performance and compare the predicted loss values for model selection. Thus, model selection is based on the predicted loss rather than on a smaller generalization bound.

Q2: "It is a bit hard for me to understand the definition of the feature vector x. What would x for GPT2 look like? What is the difference between x for GPT2 and T5? Plus, the notation x is also used in Assumption 1-3 in Section 3.1. Are they the same x?"

A2: a. In our framework, $x$ represents the features from the input space, which are used to characterize the fine-tuning data. These features encapsulate relevant properties of the input samples that influence the fine-tuning process. For specific models:

• For GPT-2, $x$ corresponds to features extracted from its auto-regressive training setup, where the inputs are sequences of tokens with causal masking.

• For T5, $x$ represents features from its encoder-decoder input format, where the model processes full input sequences bidirectionally before generating outputs.

b. The key difference is that T5 operates on a denoising objective rather than strict left-to-right token prediction like GPT-2.

c. Yes, $x$ in Assumptions 1-3 refers to the same definition as for GPT-2 and T5.

Q3: "In Equation 8, I cannot understand what x and x’ represent for. Are they some features of pre-training and fine-tuning data samples? If not, and following the definition of x as aforementioned, then why do we call a transformer model f(,)? Do we need to provide a vector $x$ with the model size, architecture, etc., to a LLM?"

A3: a. Yes, as we clarify just under Equation 8, $x$ and $x'$ refer to input feature representations from pretraining and fine-tuning data, respectively.

b. No, we do not need to provide a vector $x$ with the model size, architecture, etc., to an LLM directly. For this part, we want to provide the details of the NTK matrix that are extracted from pre-training and fine-tuning stages, and then we will pass this information to our proposed scaling law model:

$$L(D) = \frac{B}{F(\Theta, t) + D^\beta} + E$$

Overal, we are using the information from pre-training and fine-tuning stages of LLMs to help us find the transition pattern and do better prediction of performance. For the architecture of our model, please refer to the pseudo-code in section 3.2.

Q4: "The analysis of this paper heavily relies on the generalization bound provided in Ju et al. 2023 and the scaling law provided Lin et al 2024. But will they precisely describe LLM’s behavior?The experiments provided in Ju et al. 2023 only consider image classification tasks, which is quite different from LLM’s auto-regressive training. Compared with Lin et al. 2024, the experiments in this paper do not seem sufficient. IMO, extending this framework to LLMs’ finetuning needs more justifications."

A4: We would like to clarify that our work fundamentally differs from (Ju et al., 2024) and (Lin et al., 2024) as follows:

1. Framework difference from Ju et al. (2023): Ju et al. focus on image classification, but we incorporate transformer-specific elements—such as attention mechanisms, layer normalization, and residual connections—into the PAC-Bayesian generalization bound (as detailed in Appendix A), which are not addressed in Ju et al.

2. Contribution difference from Lin et al. (2024): Lin et al. empirically capture the pre-power and power phases based on heuristic scaling laws derived from observational data. In contrast, our work rigorously verifies these phases through a theoretical framework, specifically by deriving the Hessian-based PAC-Bayes generalization bound. This theoretical foundation enables us to develop a theory-grounded scaling law that precisely describe LLM behavior.

We acknowledge that our current computational resources (using a single A100-80G) limit our ability to extend experiments to larger models, and we plan to explore this in future work. Additionally, we have conducted further tests on the robustness of our method to hyperparameters; due to space limitations, please refer to the rebuttal for Reviewer 2LaB for more details.

Please let us know if there are any comments or insights, we'd like to explore further!

Thank you for the insightful questions. The followings are our answers to your concerns.

Q1: How robust is the identified transition point between pre-power and power phases to hyperparameter choices (e.g., learning rate, batch size, sequence length)? Do small changes in these settings significantly shift the transition point or degrade predictive accuracy?

A1: a. We would like to point out that the transition point is derived from the fine-tuning results as illustrated in pseudo-code in Section3.2, so its stability is closely linked to the robustness of the fine-tuning process. We conducted additional experiments on FLAN with the following setting to test the robustness of fine-tuning:

-Learning rates in (3e−5, 1e−4, 3e−4, 1e−3)

-Batch sizes in (64, 128, 256)

-Average input sequence lengths in (18, 20, 22).

Table: Variance of fine tuning results on FLAN.

The small variance ranging from 0.0012 to 0.0042 demonstrates the robustness of fine-tuning process, thereby supporting the stability of the identified transition point.

b. We illustrated the robustness of our method to hyperparameters,including regression threshold, stop threshold, learning rate, batch size and sequence length. Due to space constraints, please refer to the rebuttal of reviwer 2LaB.

Q2: For extremely large models, do you expect Hessian approximations to remain stable in practice? Or is there a risk that computing these approximations or performing partial fine-tuning becomes infeasible?

A2: We would like to point out that our approach does not rely on explicitly computing Hessian approximations for predictive modeling. Instead our scaling law prediction model is based on extracting the Neural Tangent Kernel (NTK) matrix, which effectively captures interactions between the data and model features. This allows us to make predictions about fine-tuning behavior without requiring direct Hessian computation. Thus, the Hessian is primarily used in our work to establish a theoretical Bayesian bound, which helps verify the pre-power and power phases from a theoretical perspective. In this case, we leverage general properties of the Hessian rather than computing it explicitly.

Q3: How would a significantly modified Transformer architecture (e.g., MoE, encoder-decoder) impact lens-based predictions? Would your approach require re-fitting theoretical parameters for such architectures?

A3: We would like to point out that we have evaluated both decoder-only (e.g., OPT, GPT-2, LaMini, Cerebras) and encoder-decoder (e.g., T5, mT5, BART) models without re-fitting theoretical parameters in our experiments, demonstrating the consistency of our approach across these architectures.

However, for architectures like MoE—where only a subset of parameters is active per forward pass—the effective parameter count is lower. This requires re-fitting the scaling law parameters in our NTK-based framework, specifically by adjusting the effective network width and recalibrating the scaling constants and exponents to account for the sparsity and computational cost. Due to resource constraints, we have not yet conducted experiments on MoE models, but this remains a promising direction for future work.

Q4: What is the exact set of models used? It appears that the models tested are generally not very large—could you clarify why? Additionally, do any of the fine-tuning results come directly from the rectified scaling law, or were they trained independently? If the latter, what are the fine-tuning details, including software and hardware specifications?

A4: a. As shown in Table 2, the following is our model set:

b. Due to limited access to high-end GPUs (we only used a single A100-80G), we were unable to extend experiments to larger models.

c. The fine-tuning results are all trained by ourselves, and the followings are the software and hardware details:

1. Software: Fine-tuning was conducted using PyTorch with the Hugging Face Transformers library. We use AdamW optimizer and weight decay as 0.01.
2. Hardware: Experiments were conducted on a single A100-80G.

We will clarify the above points in the revised manuscript.

A5. We would like to clarify that replacing WMT19 with Wikitext is due to computational constraints, as it is 20 times larger than FLAN and Gigaword. Additional experiments conducted on a randomly selected 1/20-size subset of WMT19 yielded results consistent with our original results.

Table: Model selection performances of our model and Rectified Scaling Law

Please let us know if there are any comments or insights, we'd like to explore further!

Thank you for your insightful questions. The followings are our answers to your concerns.

Q1: Could you please clarify the architecture of your regression model?

A1: As illustrated in Section 3.2, our regression model is constructed based on the NTK matrix as follows:

$$L(D) = \frac{B}{F(\Theta, t) + D^\beta} + E$$

where

-$F(\Theta, t)$ is the adapted NTK-based test loss function on transformer

-$D$ is the number of training data

-$\beta$ denotes the learning difficulty

-B adjusts the initial test loss

-E denotes the optimal loss of the model given an infinite amount of data.

They are all model/task-dependent and we estimate ${B, E, \beta, t}$ for each model by minimizing the loss function:

$$\min_{B,E,\beta,t} \sum_{ i} \text{LSE}(\log B - \log(F(\Theta, t) + D_i^\beta), \log E) - \log L(D_i))$$

Where $L(D_i)$ denotes the test loss of fine-tuning on the data size $D_i$, and LSE denotes the log-exp-sum operator.

Q2: Is it feasible to train the regression model on one dataset and test it on another?

A2: In our current approach, we leverage the NTK-based Rectified Scaling Model to identify a transition point in the fine-tuning process, after which we perform regression on the loss trajectory. This regression model is then used to predict performance which assist in model selection, particularly in resource-constrained scenarios.

Cross-dataset validation, however, poses a significant challenge. The primary difficulty is that the transition point identified on the training dataset may not align with that of a test dataset, due to differences in data distributions and task characteristics. This discrepancy introduce additional challenges to theoretically understand the scaling behaviors of LLM in fine-tuning. This is beyond the scope of this work, and we would like to leave it as our future work.

Q3: Could you please add Rectified Scaling Law performance in Figure 4.

A3: We would like to note that we have already included the performance of the Rectified Scaling Law in Figure 4 (https://anonymous.4open.science/r/LENSLLM-3B1E/Revised%20plot4.png) and will ensure that in the revised manuscript.

Q4: Could you discuss whether your proposed method remains effective when the test model is not included in the training set?

A4: While our experiments primarily focus on evaluating performance within the same set of candidate models, the NTK foundation capturing dynamic behaviors of LLMs during fine-tuning suggests its generalization ability to unseen models, especially those with similar architectural properties and scaling behaviors. To validate this, we conducted additional experiments on the FLAN dataset using LaMini-GPT-774M, GPT2, and BART-large as test models, with LaMini-GPT-124M serving as the training model.

Table: RMSE between predicted and actual test losses

The results indicate that our method remains effective when the test model shares similar architectural properties and scaling behaviors with the training model (e.g., LaMini-GPT-124M vs. LaMini-GPT-774M and LaMini-GPT-124M vs. GPT2), which is further supported by our theoretical foundation. Please let us know if there are any comments or insights, we'd like to explore further!

Q5: Could you provide some discussion or how to extend your method to vision-language model selection?

A5: Our current approach models fine-tuning dynamics using the NTK matrix and scaling laws to capture training dynamics and Hessian properties in language models. Extending this framework to vision-language models presents several challenges:

1. Cross-Modal NTK Formulation: The NTK must be adapted to capture interactions between tokenized image representations (e.g., VQ-VAE or CLIP features) and textual tokens, reflecting joint feature spaces and inter-modal attention.
2. Modality-Specific Scaling: Vision and language modalities have different scaling behaviors, requiring recalibration of the scaling laws to account for distinct gradients and effective parameter counts, as well as synergy and competition between modalities(Armen Aghajanyan et al., 2023).
3. Theoretical Bound Adjustments: Our current PAC-Bayesian and NTK-based bounds are tailored to language models; for vision-language models, these would need to be re-derived or adjusted to include modality-specific properties.

Due to these complexities, extending our method to vision-language model selection is non-trivial and beyond the scope of this work, but it remains a promising direction for future research.

Please let us know if there are any comments or insights, we'd like to explore further!

Q1: How sensitive is the method to hyperparameter choices, particularly in the algorithm's stopping criteria?

A1: Thank you for your question.

a. We performed ablation studies to assess the sensitivity of our method to the stopping criteria—specifically, the regression threshold ($\gamma$) and the stop threshold ($\tau$). The following tables summarize the impact of varying these parameters on the Pearson correlation across three datasets:

Table1: Impact of $\gamma$ and $\tau$ on PearCorr on FLAN

Table2: Impact of $\gamma$ and $\tau$ on PearCorr on Gigaword

Table3: Impact of $\gamma$ and $\tau$ on PearCorr on Wikitext

Observations:

1. FLAN: Pearson correlations remain stable (approximately 78.31 to 78.43) across different values of $\gamma$ and $\tau$.
2. Gigaword: The correlation values are consistent, ranging from about 85.66 to 85.79.
3. Wikitext: A slight variation is observed, with correlations fluctuating between roughly 82.42 and 82.61.

Overall, the minor fluctuations in Pearson correlation across different $\gamma$ and $\tau$ values indicate that our method is robust concerning the stopping criteria.

b. We also conducted some additional experiments to test the sensitivity to hyperparamters in fine-tuning process - specifically, learning rates, batch sizes and average input sequence length.

The impacts of learning rates and batch sizes on the Pearson correlation across three datasets are summarized below:

Table4: Impact of Learning Rate and Batch Sizes on PearCorr on FLAN

Table5: Impact of Learning Rate and Batch Sizes on PearCorr on Gigaword

Table6: Impact of Learning Rate and Batch Sizes on PearCorr on Wikitext

Observations:

1. FLAN: Pearson correlations remain stable (approximately 78.32 to 78.43).
2. Gigaword: The correlation values are consistent, ranging from about 85.69 to 85.79.
3. Wikitext: A slight variation is observed, with correlations fluctuating between roughly 82.50 and 82.61.

Overall, our method is robust concerning the learning rates and batch sizes.

Due to time constraints, we evaluated only the average input sequence length on FLAN while keeping the optimal learning rate (3e-4) and batch size (128) as determined earlier. In FLAN, the overall average input sequence length is 20. To test the effect of altering this average, we removed either the shortest or longest sequences to adjust the average to 18 and 22, respectively.

Table7: Impact of Average Input Sequence Length on FLAN

We observe that the deviations—either shorter (18) or longer (22)—lead to lower Pearson correlation and relative accuracy, the performance gap is not large, which suggests that while there is some sensitivity to sequence length, the model remains reasonably robust.

Please let us know if there are any comments or insights, we'd like to explore further!