Learning Dynamics of LLM Finetuning

Thank you for your great comments and efforts in reviewing our paper. Please also refer to the overall response, in addition to the response below.

1. Need a "related work" section. …

Thanks very much for pointing this out. Please also refer to the third section in the overall response. Due to the page limits, we added only the related work section in Appendix E (the last appendix in the original version). We agree with the reviewer that offering a more thorough related work section would help ICLR readers better. The new related work section will not only cover learning dynamics and LLM finetuning but also mention that the “peakiness effect” was previously observed (see the third point of the response to the reviewer 4pPy), where we find their experiments match our analysis well.

2. Need to indicate assumptions in your Prop 1 and Section 3.2 clearly …

As the assumption we made on the eNTK term is mentioned by all the reviewers, we explain this in the first section of the overall response, and will make this prominent in the revised paper.

The statement of Proposition 1, however, is fully accurate; the lower-order terms are unambiguously $\\mathcal O(\\eta^2)$ as in the statement. Since the learning rate is usually quite small during finetuning, over a short time period the first-order effects should dominate; over a longer time period, other factors can grow important. This is in fact the same concern as in the previous section; if we really followed only the first-order terms, the eNTK would remain constant. In adding our discussion of the previous point, we will make sure to emphasize this.

3.a Should design more experiments … During off-policy DPO, show that no other sentence's log probability significantly increased …

Unfortunately, the Y space is far too large for us to evaluate the probability of all other responses as we did for the MNIST experiment – which is probably why it was not previously noticed that it was the greedy decoding that took most of the increase during DPO. Towards verifying this, however, we will add the following discussions and experiments in the next section (details in Figure F.4 in the rebuttal appendix).

For the greedy $y^*$, we also track the “except-argmax” probability, i.e., besides tracking $\\pi(y^*\|x\_u)=\\sum_l p(y^*\_l|y\_{<l}^+)$, we also track $\pi(y^*|x\_u) ^C\triangleq\sum\_l p(y\neq y\_l^*|y\_{<l}^+)$ and compare them in the same figure. It is clear that $\\pi(y^*|x\_u) ^C$ decreases as $\\pi(y^*|x\_u)$ increases. The important thing is that for DPO, we find $\\pi(y^*|x\_u) ^C$ ends up much smaller than in the SFT case. Note that this term is calculated by summarizing all other $V-1$ dimensions. In other words, during the late stage of DPO, for each token’s prediction, the model’s confidence in each token is greater than 0.5, which makes it unlikely for the existence of other high-likelihood responses.

Furthermore, we can also infer the non-existence of higher-confidence sequences by thinking one step further from our original results. Recall we mentioned that in the DPO case, the log probability of $y^*$ is approaching -63 in Line 406. Since the average response length of $y_u^+$ is 261 in our probing dataset, the model’s average confidence on each token is roughly $e^{-63/261}\\approx 0.79$, much highher than 0.5. Hence we believe it is unlikely to have to have other sequences with significantly higher confidence than $y^*$.

3.b For SFT, quantitatively show the causal effect of increased hallucinations due to …

Thanks very much for this suggestion. We first apologize for the inadvertant over-claim in our original version that the “pull-up” effect explains why hallucination occurs. What we want to claim is a hypothetical explanation of a specific type of hallucination, i.e., using the answer or phrases in question B to answer question A, since the confidence on $y\_{j\neq i}$ keeps increasing during the SFT stage. We absolutely do not claim that this is representative of all or even most types of model hallucinations. We looked at related work on quantitatively analyzing the hallucination effect, e.g., [1], but those we read rely more on evaluating whether the answer provided by the model comes from the data or its internal information. The hallucination type we mentioned is more about whether the answer provided by the model wrongly mixes the facts provided by the dataset, which is a bit harder to directly observe. We think learning dynamics and our particular results might provide a useful jumping-off point for further exploration of this important effect, but it certainly does not trivially solve the problem.

(...To be continued)

Thank you for your great comments and efforts in reviewing our paper. Please also refer to the overall response, in addition to the response below.

1. The first is the obvious limitation that the entire analysis is being done under the assumption that the “feature map” is held fixed …

Thanks very much for this question, which helped us realize what the assumption we actually need is. Please see the first point in the overall response, which thoroughly discusses the core assumption we need (which is far weaker than that the feature map is fixed). Furthermore, in the rebuttal appendix (Appendix F), we derived two metrics estimating the lower bound of the K term, and tracked their stability instead. In calculating them, we directly track $\\|A\_o^t\\|\_F$, $\\|G^t\_u\\|\_F$, $\\|\Delta\_o^t\\|\_F$, and find that they all match our analysis well. Although we only plan to add experiments on these two metrics in the next version due to the page limits, we can provide the details of other intermediate terms if it would be helpful.

2.1 The second limitation is more nuanced, and has to do with the relying of some intuitive understanding of "similarity" …

This is a great point that we had to think seriously about. The relative strength of the eNTK is an objective quantity which can (at least conceptually) be computed, and it is indeed (up to a first-order approximation) exactly what “similarity” means to the current model's gradient updates. How thoroughly this aligns with our human intuition about “similarity”, however, is much less clear, and some of our discussions certainly relied on intuitive notions of similarity rather than measuring the eNTK. This measurement is easier for the MNIST example, as the CNN extracts semantic information fairly similar to our intuition, and we find that “4” and “9” are similar in both intuition and experiments. For language data, on the other hand, there are many aspects of similarity that can easily be task-relevant. For instance, two sentences can be similar in their meaning or in their form. This is why we created two rephrases by ChatGPT to try to find some hints. The original Figures 3 and 4 verify our intuition that natural language is dissimilar to random sequences, the chosen response and its rephrases are more similar than the rejected ones, etc. The fact that the LLM relies more on semantic similarity is also mentioned in related work.

In our new experiments in Appendix F.2 (inspired in large part by this question), though, we discovered a new similarity in “where the sequence is generated”. In Figure F.3, it is clear that learning examples generated from ChatGPT influence more on those rephrases than its original response. A fuller intuitive understanding of similarity as defined by the eNTK remains an open problem; we plan to explore this further in the future.

2.2 For example, "the model’s confidence … (Line 350)

This is another very good point. As demonstrated by Figure F.5-(b) in Appendix F, we see that the increase of $\\pi(y\_u^+)$ does indeed slow down a bit after the 2nd epoch; this is difficult to see by eye, but holding a piece of paper up to the endpoints makes it clear there is some curve to it. Furthermore, it is true that from the curve of $\\log\\pi^t$ (as in the figures in the original version), we cannot see a clear drop in the updating energy. That is perhaps because the term $A\_o^t$, whose Frobenius norm is $\\|A\_o^t\\|\_F = V \\|\pi^t\\|\_2^2 + V - 2$, will also gradually increase as $\\pi^t$ becomes peakier (see the derivation in Appendix F.2). Note that the 2-norm of a one-hot distribution is one, while the 2-norm of a uniform distribution is $\\frac{1}{\\sqrt{V}}$. So, even though the upwards pressure indeed decreases, it is partially mitigated by the increase in $\\|A\_o^t\\|\_F$.

(...To be continued)

I thank the authors for their detailed response. I think the revisions proposed in the new rebuttal appendix contribute to a better understanding of the paper (and Fig 1 is definitely improved). The issue of what "similarity" is persists, but as the authors (rightly) mention -- this is, in effect, a larger open question for the field. And methods of the form proposed in this paper might eventually contribute to a better understanding of that question. Following the authors response I have raised the "soundness" score to 4, and overall I stand by my evaluation that this is a good paper and should be accepted.

Thank you for your great comments and efforts in reviewing our paper. Please also refer to the overall response, in addition to the response below.

How does the squeezing effect differ from overfitting with DPO?

Thanks very much for this insightful question. We think in both the squeezing effect and overfitting, the model becomes more confident on some predictions, i.e., the predicting probability becomes peakier. However, the main difference between them is which dimension they are converging to. From the first panel in Figure 2, overfitting will make the model remember all the details (even noise) in the dataset, and hence peak at the dimension of the supervisory signals (usually, the ground-truth labels). The squeezing effect, on the other hand, makes the model amplify its own current beliefs, whatever they may be, and hence peak at the dimension with the highest output prior to the update step (this is guaranteed to happen). Furthermore, as analyzed in Appendix C, the squeezing effect will decrease the probability mass on all the dimensions other than the most confident one, which means the catastrophic forgetting in the squeezing effect might be more serious than in overfitting. We believe this is also an interesting direction to explore in the future.

Thank you for your great comments and efforts in reviewing our paper. Please also refer to the overall response, in addition to the response below.

1. The clarity of Sec 3.1 when discussing the causal masking …

Thanks for pointing this out. We have added a figure demonstrating the causal masking and a short paragraph to explain this point more clearly; it is currently in Figure F.5a, and will be merged into the main paper in the next version.

2. Additionally, while the analysis relies on the kernel values between the feature maps …

Thanks for this great question. Since this concern also aligns with other reviewers, we discuss this point in the overall response. Specifically for your concern, in Appendix F.2, we derived two metrics estimating the lower bound of the K-term and tracked their stability instead. In calculating them, we directly track $\\|A\_o^t\\|\_F$, $\\|G^t\_u\\|\_F$, $\\|\Delta\_o^t\\|\_F$, and find that they all match our analysis well (Figure F.3 and F.4). Although we only plan to add experiments on these two metrics in the next version due to the page limits, we can provide the details of other intermediate terms if it would be helpful.

3. Can you discuss the analytic or empirical argument for "Peakier …

Thanks very much for pointing this out. It is true that the claims about the relationship between “how peaky $\pi$ is” and “how serious the squeezing effect is” (Claims 3A and 3B) are based on intuition and numerical simulations on equation (27)-(32). 3A is more formal because we can substitute the definition of uniform distribution for these equations, and find deterministic relationships. We will make the distinction between fully-analytical statements and those stemming from empirical results clearer in revision.

4. The color coding of the variables in section 2 and 3 …

Thanks for pointing this out. The coloring system indeed becomes more chaotic after section 3.1, as more notations (especially about DPO) are introduced. Our motivation for using these colors is to remind the readers that the orange thing is the observation input and the blue thing is the updating input. However, this color encoding contradicts those used in Fig.2 and other experimental results (e.g., chosen and rejected responses). In the next version, we plan to only use these two colors in section 2 (and the appendix), and change the others back to black.

(apologies for the delay in responding)

I appreciate the authors' efforts to add the additional diagram into a final draft, and the analyses for intermediate values in the influence metrics. I think that the F.3 panel is quite interesting and so maybe a concise summary chart with a few curves to be highlighted could be worked into the main body (or at least noted in poster/talks etc).

Assuming inclusion of the other minor adjustments in terms of precise organization of claims and notational polishing in the main body discussed in this specific rebuttal, and in the general discussion with all reviewers, I think that this work is of high overall quality and potential impact, and is appropriate for the ICLR venue. It should be considered for highlighting at the conference.

Specifically, in almost all the training curves demonstrated in [2], we observe a slow-change norm and relatively stable learning dynamics in the latter phase, which also aligns with Figure 6 in [3], which measures the eNTK change in an image classification problem. In summary, although it is hard to directly calculate the huge NTK matrix for the GPT-based model (>1.0B params), likely, the relaxed version of the relative-lazy-NTK assumption (i.e., the one we used) holds for different settings. We will also indirectly verify this using experiments later.

2. More about the evaluation of the repeating effect

In the next version, we also add more results showing the relationship between repeatness, DPO, and the proposed method. Specifically, we use ChatGPT and Claude API as a judgment to count how many “non-human-like-repeated ” responses exist in the model’s response. Although we only show the example of one run in one setting, some vague trends still exist here. In the following table, we see the number of non-human-like-repeated indeed increased as the DPO continues (after SFT → DPO 2 epochs → DPO 6 epochs). Comparing the proposed method and baseline, the "repeater effect" is indeed slightly reduced. However, due to time constraints, we haven’t tracked whether false positive and false negative judgments exist (we believe there might be some). We would leave the more detailed experiments in our future work.

3. Limitations and related work discussion

In the next version, we will put some experimental results in the appendix and add a more thorough related work section (a version is now in Appendix E, but we will indeed add more discussion), as well as further discussing the limitations of our theory.

One important update is about the “benign peakiness” effect mentioned in [5-7], which strongly supports our discussions on the “squeezing effect”. The discussions in the previous work confirm our claim that the negative gradient is one important cause of the squeezing effect. For example, Equation 1 in [5] shows that we minimize a negative thing in a standard GAN loss, which might explain why peakiness occurs. Furthermore, in Table 1 and Table 2 of [6], we see the peakiness (measured by $\Delta p_{top10}, \Delta p_{mode}$) of the “PG-average” method is stronger than the standard PG method. Note that the “PG-average” method will map a reward ranging from 0 to 1 to a centered one ranging from -0.5 to 0.5. Since the negative reward can introduce a negative gradient, the peakiness increases.

However, we would also emphasize that the “harmful squeezing effect” discussed in this paper has another important factor. As demonstrated by our Figure 16: imposing a negative gradient on a highly-likely point only introduces a mild adaptation, while one on a very-unlikely output leads to drastic squeezing. In on-policy DPO and the scenarios of [5-7], the negative losses are imposed on a point generated by the current model, hence one which is reasonably likely; thus the peakiness effect in those settings is not very harmful, or even beneficial. The method we proposed in section 4.3 exactly tries to convert a harmful peakiness effect to a benign or beneficial one by “pulling up” the valley before DPO.

[1] Malladi, Sadhika, et al. "A kernel-based view of language model fine-tuning." ICML 2023

[2] Li, Ming, et.al "What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective." arXiv 2024

[3] Ren, Yi, et al. "How to prepare your task head for finetuning." ICLR 2023

[4] Fort, Stanislav, et al. "Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel." NeurIPS 2020

[5] Caccia et al., Language GANs Falling Short, ICLR 2020

[6] Choshen et al., On the Weaknesses of Reinforcement Learning for Neural Machine Translation, ICLR 2020

[7] Kiegeland and Kreutzer, Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation, NAACL 2021