The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation

Thank you for reviewing our work and recognizing the appeal of the hyperfitting phenomenon and our presentation of it. We appreciate your feedback and hope you find the following arguments and revisions to increase your rating.

In regards to the paper’s contribution:

We do not interpret ICLR to be a conference aimed solely at applied science. Indeed, we argue that a fundamental phenomenon that recurs across various models and modalities is a great fit for this conference, especially as you find our evidence conclusive and the phenomenon itself interesting.

While pushing LLM capabilities on comparable benchmarks is a worthy endeavour, it is not the only frontier to explore. Our discovery instead pertains to the underlying learning dynamics occurring within the models—something we argue to be of equal relevance due to the prevalence of next-token prediction, perplexity loss, and the repetitive nature of greedy decoding. Our intention is not to propose a new method but to share the existence of this counterintuitive yet reproducible and widespread phenomenon. Hence, we see this as a great way to further the scientific understanding of the mechanisms of these networks.

In regard to additional sampling methods:

Although we did involve a baseline using Top-P sampling in Table 1 (line 169) – which hyperfitting outperforms in human preferences – we emphasise that we are not looking to propose a practical method to increase LLM capabilities.

Instead, our intention is to shine a light on the counterintuitive hyperfitting phenomena, and how it improves human evaluation scores, despite drastically worsening metrics such as perplexity. The current version of the paper may appear to focus too much on the significant improvements in human evaluation scores, which could give the impression that hyperfitting is a practical training method. We are very willing to clarify this and emphasize how our discovery provides a counterintuitive insight into large neural networks. Would such a clarification in the paper's narrative sway your judgement of the paper’s contribution?

In regards to the root cause of hyperfitting:

Understanding the root cause of this phenomenon has proven challenging due to its counterintuitive nature. Several investigative experiments were conducted to investigate how hyperfitting alleviates the repetition problem, but we were unable to achieve conclusive results. Would including a list of failed exploratory experiments enhance your evaluation of the paper’s contribution?

In regard to performance on downstream tasks:

Several reviewers asked about hyperfitting's impact on benchmark performance. Appendix B.3 shows results on the GLUE benchmark, revealing a surprising finding: hyperfitting only slightly reduces scores. As reviewer gnwy noted, models with poor perplexity loss would typically be expected to perform near-randomly on such tasks.

Since MMLU results were specifically requested, we will include them with comparisons to instruction-tuned and chat models. While referencing these in the main text is possible, space constraints may require moving another experiment to the appendix if you think this is of great importance.

Direct responses to questions:*

1. Lines 284 and 301 both refer to the finetuning dataset, which we used for hyperfitting. We acknowledge the potential for confusion and will clarify this.
2. The finetuning dataset is most likely a subset of the training datasets, though we cannot confirm definitively as we lack a full list of the pretrained models' training data.
3. We expect generation results in your proposed random decoding experiment to closely resemble greedy decoding due to low entropy. Hyperfitted models produce more deterministic outputs than non-hyperfitted ones, large adjustments to temperature non-withstanding.

Thank you for taking the time to review our work.

We are glad to hear that you find our paper well-written and convincing regarding the existence of hyperfitting. If we understand your feedback correctly, your low rating is based mainly on our discovery seemingly lacking a direct practical application. We would gladly hear your thoughts and potential changes in your rating, given our arguments and proposed revisions.

In regard to the lack of applications:

We do not interpret ICLR to be a conference aimed solely at applied science. Indeed, we argue that a fundamental phenomenon that recurs across various models and modalities is a great fit for this conference, especially as you find our evidence conclusive and the phenomenon itself interesting.

While pushing LLM capabilities on comparable benchmarks is a worthy endeavour, it is not the only frontier to explore. Our discovery instead pertains to the underlying learning dynamics occurring within the models—something we argue to be of equal relevance due to the prevalence of next-token prediction, perplexity loss, and the repetitive nature of greedy decoding.

Our intention is not to propose a new method but to share the existence of this counterintuitive yet reproducible and widespread phenomenon. The current version of the paper may appear to focus too much on the significant improvements in human evaluation scores, which could give the impression that hyperfitting is a practical training method. We are very willing to clarify this and emphasize how our discovery provides a counterintuitive insight into large neural networks.

Would such a clarification in the paper's narrative sway your judgement of the paper’s contribution?

In regard to the lack of LLM benchmarks:

Several reviewers have requested information about how hyperfitting impacts a model’s benchmark performance. Appendix B.3 contains results on the classical GLUE benchmark, where we observe another counterintuitive result: hyperfitting only marginally decreases the score. As reviewer GNWY pointed out, one would expect a model with extremely poor perplexity loss to perform near-randomly on these tasks.

Since the MMLU test has been specifically requested, we will aim to add scores for this as well before the deadline. However, we note that the absolute performance will likely be far from SOTA, as the models are not instruction-tuned. We will reorganize the main paper to directly reference these results. Including them in the main body might be challenging given the current paper limit, but if you think this is of great importance, we could replace another experiment and move it to the appendix.

Thank you for your encouraging review and helpful tips for improvements.

We fully share your thoughts on how this phenomenon opens up a new perspective in understanding repetition in text generation. Rest assured, your feedback does not fall on deaf ears, and we will incorporate as much as we can during the rebuttal period (see below). Given that your feedback is incorporated, would these additions prompt a potential increase in your rating?

In regard to larger models:

To fulfil your request, we will attempt to hyperfit a 70B Llama 3.1 model. While it is uncertain whether we will be able to collect human annotation data in time, we will try our best. At the very least, we should be able to provide some automatic metrics by the end of the rebuttal period.

Intuitively, one would expect the hyperfitting phenomenon to extend to larger models, though the difference compared to pre-hyperfitting is likely to be smaller. This aligns with our reasoning regarding top-rank encouragement, as larger models typically achieve a lower training loss. This may imply that achieving similar behaviour might not require an equally large model.

In regard to ambiguous implementation details:

Thank you for pointing out this lack of clarity. In all our experiments, we train the full model and update all parameters. We will make changes to Section 3 to clarify this further.

We share your enthusiasm for exploring hyperfitting with LoRA and other methods. It would indeed be fascinating to investigate how many parameters can be frozen while still invoking this phenomenon. However, we would argue that such research would be more suitable for future work and is not a weakness of the current paper.

In regard to limited datasets/tasks:

This is an excellent suggestion, especially if we can rely on automatic metrics such as BLEU. We intend to run experiments on the datasets you have proposed.

Thank you for your thorough review. We are delighted that you find the phenomenon interesting and the overall presentation to be of high quality.

We hope you consider our arguments below, and we are more than happy to hear your thoughts. Thank you for your useful feedback, particularly regarding your proposal to hyperfit already instruction-tuned models.

In regard to lack of utility and benchmarks:

We do not interpret ICLR to be a conference aimed solely at applied science or utility. Indeed, we argue that a fundamental phenomenon that recurs across various models and modalities is a great fit for this conference, especially as you find our evidence conclusive and the phenomenon itself interesting. Our intention is not to propose a new method but to share the existence of this counterintuitive yet reproducible and widespread phenomenon.

With that in mind, Appendix B.3 contains results on the classical GLUE benchmark, where we observe another counterintuitive result: hyperfitting only marginally decreases the score. We share your intuition that a model with such extremely poor perplexity would be expected to perform near-randomly on these tasks. However, this is evidently not the case.

Since MMLU has been specifically requested, we aim to include this test as well. As you note, however, the absolute performance of our models will likely be far from SOTA, particularly since the models are not instruction-tuned. Nonetheless, as discussed in the next section, instruction-tuned models themselves represent an interesting addition to the paper.

In regard to only testing base models:

Our focus on base models is due to hyperfitting mitigating the repetitive nature that arises from next-token prediction, a phenomenon that occurs in both text and image models trained solely using this objective. In contrast, instruction-tuned models often incorporate more complex training algorithms, such as various forms of reinforcement learning.

That said, we share your interest in how hyperfitting might affect instruction-tuned models. Therefore, we will strive to include experimental results analyzing its impact on LLM datasets such as MMLU, both pre- and post-hyperfitting. Extrapolating from the findings in Appendix B.3, we expect the change in performance to be less significant than one might initially anticipate.

In regard to token diversity and textual quality:

We agree that a high TTR does not guarantee high textual quality. This is why we relied on extensive human annotation data. However, as specified in Appendix A.1, a low TTR correlates with low textual quality.

This section is referenced at line 131:
“…and is further discussed in Appendix A.1.”
And at line 391:
“Further discussion regarding TTR as an automatic metric is available in Appendix A.1.”

Please let us know exactly where we have implicitly or explicitly stated that TTR directly equates to textual quality so that we can rectify any overstatement. Any such feedback would be much appreciated.

In regard to minor weaknesses:

These are excellent suggestions, and we will clarify the text as you propose. Regarding @1/3/5 Prob, the notation is borrowed from recall, which we agree is not very clear here. Specifically, @1 represents the probability of the top candidate, @3 the combined probability of the top 3 candidates, and so on. This table, in essence, illustrates how sharp the hyperfitting distributions are.

(apologies for the delay in responding)

Firstly, I am aware of the broad scope of ICLR. Rather my commentary about a narrow evaluation is a face value statement wondering about "who will glean useful insights from the work?" rather than any arbitrary value judgement about fit for a big tent ML conference.

TTR

I appreciate the note about the appendix section discussing TTR.

Although TTR is a simple metric measuring the ratio of unique tokens,

... is just not the most clearly worded sentence, and TTR appears in the Fig. 2 caption and axes without the definition. To a casual reader skimming the work, this would limit what they can clean in first pass from the figure which is quite self explanatory if the axis instead said Generation Diversity, or Unique 1-grams (%) or something else at least partially self-evident. Treat this comment as a failure of this reviewer to know the acronym, but accept that this is likely also true for many readers in the (quite broad) potential audience for ICLR papers on the science of LLMs.

MMLU

The additional evaluation of Instruction Models and MMLU performance is greatly appreciated and adds value to the draft. Ideally more modern tasks than just MMLU would be surveyed, but the agreement between the trends observed under hyperfitting between GLUE and MMLU is already clear.

It is indeed surprising that the that while assigning astronomical perplexities to general text, the ability to assign ordered likelihoods to candidate answer choices is still nontrivially maintained.

Could the authors please add the Perplexity on the actual answer choices for the benchmark to Table 6? Say, the avg. likelihood of the correct response, just to get a sense of whether the scores assigned to MMLU or other benchmark answers are as elevated as the general text PPLs are after hyperfitting?

Further interest in Chat model analyses

Additionally, did the authors also compute some of the same measures from for instance Table 1,2, or 3 on the Chat versions of the Deepseek and Llama models?

While I appreciate the inclusion of the Chat models in the MMLU analysis (which is a check for degradation), my real interest in the review section Questions Primary 2.) was trying to gain insight into whether any of the improvements in generation characteristics are observed when hyperfitting already instruction tuned models, or whether it has no effect in this case, or causes generation behavior (say TTR only for practical reasons) to otherwise change in any measureable way.

Current recommendation

Generally I appreciate the effort of the authors in addressing my questions so far, and am already more confident in the contribution of the draft given the updates discussed so far, so I will increase my score.

If the authors were already able to compute any of the main body metrics for the chat models as well (Tables 1,2 or 3), I think this could add to the contribution and impact further since the hyperfitting phenomenon itself is demonstrably interesting, but now needs to be mechanistically unpacked to provide further actionable insights. As a starting point for this work, I believe that understanding the degree to which taking a base model and hyperfitting it versus applying regular post-training (ie. its Chat variant) produce different effects, is part of this process.