Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs

Thank you for reviewing our paper and your detailed comments! Please find our responses to the concerns raised. Please let us know if you have any further concerns or suggestions.

The release of the activeft library as a drop-in replacement for Nearest Neighbor retrieval supports transparency and facilitates future research in prompt-specific fine-tuning methods.

Thank you for highlighting this!

The motivation and definition of the task lack clarity, particularly in distinguishing test-time fine-tuning from standard fine-tuning on selected data. It remains unclear why test-time fine-tuning is necessary in this context and how it fundamentally differs from simply fine-tuning on pre-selected data, which may impact understanding of the novelty and importance of SIFT’s approach.

We added a number of additional results on state-of-the-art models in our revised paper, and added Figure 7, showing the absolute performance gain of test-time fine-tuning over the base model. Please see our main official comment for more details. We hope that our inclusion of additional results showing the large absolute performance gains that can be achieved by test-time fine-tuning of a base model (even on "in-distribution data" such as the Pile), further supports the setting of `"learning at test-time". It seems interesting to us that test-time fine-tuning improves the performance of a state-of-the-art model like Phi-3 (3.8B) much more than going to the largest Phi-3 (14B) model. We make the same observation with Llama-3.2 models.

We see test-time fine-tuning as the extreme end of a spectrum, with pre-training (where we aim to fit the entire data distribution) on the opposite end. Loosely speaking: In the same way that we would expect a model fine-tuned on coding to do better on competitive programming than the base model, we would expect a model fine-tuned on competitive programming to do better on competitive programming than the model fine-tuned more broadly on coding. Within this example, our work studies models fine-tuned to a specific competitive programming task. We would expect that such a fine-tuned model tailored specifically to the task at hand can perform outperform a model that is learning to "solve all competitive programming tasks at once" since it has to use its weights only to encode the task-specific information, while being able to ignore all other information. For example, recently, test-time fine-tuning has been applied in state-of-the-art methods for the ARC challenge [e.g., 1].

Thank you for reviewing our paper! You can find below our response to your questions and concerns. Please let us know if you have any further concerns or suggestions.

Organization and Insight: It is well-organized and self-consistent, providing thorough discussions on the research topic and outlining both current and future research directions.

Thank you! We are glad to hear that you enjoyed reading our paper.

Inference Cost: Comparison with Nearest Neighbor, this method may require more inference time. It would be beneficial to compare inference times with Nearest Neighbor across different datasets to quantify this.

Thank you for highlighting this! We conducted a comparison in Figure 4, which shows the compute overhead of using SIFT as opposed to Nearest Neighbor (NN) retrieval. These results are averaged across all test examples, with bootstrap error bars. While SIFT costs slightly more than NN for retrieval, relative to the combined cost for retrieval and training, SIFT costs essentially as much as NN (i.e., incurs almost no additional overhead). We hope this addresses your question adeptly.

Broader Evaluation: The paper could explore effectiveness on more datasets and larger models, such as LLaMA-3, to validate its scalability and generalizability.

The Pile dataset is a huge meta-dataset that contains a large variety of datasets, each of which testing different domains of human language. These include web text, Q&A, Wikipedia, scientific papers, code, math, court opinions, patent applications, emails, and numerous more. We therefore believe that the Pile dataset offers a very rich testbed for evaluation of language modeling performance. Nevertheless, we agree that it would be interesting to evaluate SIFT on downstream non-perplexity tasks, and we are inclined to leave such an evaluation to future work.

Regarding the tested models, we added additional evaluations on Llama-3.2 (1B & 3B). Moreover, we evaluated the base performance (without test-time fine-tuning) of many state-of-the-art models (including Phi-3-14B, Gemma-2-27B), and find that our test-time fine-tuned models achieve vastly greater performance. We find that Llama-3.2-3B with test-time fine-tuning achieves a new state-of-the-art on the Pile language modeling benchmark. We describe these new results in greater detail in our main official comment above.

We hope to have addressed your concerns and are thankful for your recognition of our contributions. We would be happy to answer any remaining questions or concerns.

Thanks for all your responses, Addressing most of my concerns, I keep my score.

Thank you for reviewing our paper! Please find our response to your question. Please let us know if you have any further concerns or suggestions.

This paper verifies a meaningful yet little explored topic in LLM application, giving comprehensive and solid discussion on the challenge and the proposed solution. The whole paper is clearly organized and easy to follow.

Thank you!

The experimental study is convincing and covers a wide range of datasets.

Thank you!

The interpretability of this work might be improved by giving some instances of data selection.

Thank you for this suggestion! We very much agree and added examples to Appendix L.2 (end of the pdf). We believe that this makes our results easier to interpret.

How is the possibility and gain of combining this strategy with orthogonal methods of LLM finetuning?

Also thank you for highlighting this! We agree that many techniques from LLM fine-tuning might also be applicable to our setting. Indeed, our work shows that parameter-efficient fine-tuning via low-rank adaptation (LoRA) is effective for test-time fine-tuning. We believe that the evaluation of further techniques is a great idea for future work.

We are glad to see your recognition of our contribution. We would like to highlight the results added with our revision, which achieve a new state-of-the-art on the Pile language modeling benchmark (please see our main official comment above). We would be happy to answer any remaining questions or concerns!

Thank you for your review and detailed comments! You can find below our detailed response to your questions and concerns. Please let us know if you have any further concerns or suggestions.

Effective Data Selection with SIFT: SIFT combines uncertainty and diversity to select non-redundant data, enhancing test-time fine-tuning efficiency compared to traditional methods. Comprehensive Experiments: Wide experiments on the Pile dataset demonstrate SIFT’s effectiveness, consistently outperforming traditional Nearest Neighbor and other baseline methods in fine-tuning efficiency and model performance.

Thank you!

Complexity and Clarity of Method Presentation: The paper’s explanation of the SIFT algorithm, especially on pages 4 and 5, could benefit from clearer descriptions and simplification of symbols. The complex notation and detailed mathematical formulation may obscure understanding for readers, especially those less familiar with active learning or information-theoretic approaches. Providing a more accessible walkthrough or visual aids could improve clarity.

Thank you for raising this important point! We added a discussion of an example to Appendix C.1, which shows the close connection of SIFT to Nearest Neighbor retrieval. We hope that this supports the understanding for readers more familiar with the literature of retrieval-augmented generation and LLM vector search methods.

Sensitivity to Hyperparameters: The method relies on certain hyperparameters, such as the regularization parameter ($\lambda'$), which can significantly impact SIFT’s performance. Techniques for automatic tuning or guidelines for parameter selection would make the approach more user-friendly and robust.

Thank you for suggesting the tuning of $\lambda'$. We agree that this could be an effective approach to further improve performance.

We would like to emphasize that $\lambda'$ is the only additional hyperparameter of SIFT compared to other data selection methods such as Nearest Neighbor (NN) retrieval. Our experiments show that — even without any adaptive tuning — SIFT outperforms NN for any tested $\lambda'$ between $10^{-8}$ and $10^1$ (cf. Figure 5, right). Moreover, as we show in our ablations (cf. Table 5 in Appendix J), SIFT outperforms or performs on-par with NN for all such $\lambda'$ on any of the diverse individual datasets of the Pile. This suggests that SIFT is robust to the choice of $\lambda'$, and that any "reasonable" choice leads to strong performance. We added a summary of the above to "Insight 2" in Section 5.

Relevance and Diversity. It is strange to mention that "we provide an example of how SIFT balances relevance and diversity, where we also see that the parameter"

Thank you for pointing this out. What we meant to say is that there exists a tradeoff between selecting the most relevant data (i.e., repeatedly selecting the nearest neighbor) and selecting diverse data (i.e., where each data point is as dissimilar as possible to all other data points). The parameter $\lambda'$ allows to smoothly interpolate between both extremes. We added Figure 9 (left) in Appendix C to illustrate this tradeoff and how it is governed by $\lambda'$. The performance implications of this tradeoff are visualized in Figure 5 (right), where SIFT converges to NN-F as $\lambda' \to \infty$ and to "purely maximizing diversity" of the selected data as $\lambda' \to 0$. These experiments show that not choosing either one of these two extremes (i.e., not choosing $\lambda' \to 0$ or $\lambda' \to \infty$) can vastly improve performance, robustly across a large range of $\lambda'$-values.