Test-Time Learning for Large Language Models

We are deeply grateful for your thoughtful and encouraging feedback. Your recognition of our motivation and the thoroughness of our experiments is truly inspiring. Our detailed responses are as follows:

Q1. One experiment question is that how does your methods compared with other adapting methods such as fine tuning and rag.

A1. Thank you for the insightful question. Our work focuses on TTL, which adapts LLMs using only unlabeled test data during inference. In our experiments, we thus chose to compare our method with other TTA methods, which share a similar problem setting. We would like to clarify further the rationale for not including fine-tuning and RAG-based methods in our experimental comparison, along with the potential advantages of our method:

• Supervision and Data Dependency. Fine-tuning requires labeled data, which is costly and often impractical in deployment. RAG depends on a well-maintained retrieval corpus, which may not generalize well across domains. In contrast, our method uses only test data and minimizes perplexity to adapt efficiently without labels or external knowledge.
• Experimental Design Rationale. Our experimental design is deliberately aligned with the TTL setting, where no labels or external knowledge sources are available at deployment. Therefore, we select baselines (e.g., Tent, EATA) that also operate in this setting. Although fine-tuning and RAG are powerful in their respective contexts, they fall outside the scope of our setting due to their dependency on supervised data or retrieval infrastructure.

Q2. Reproducibility: Missing details (e.g., random seeds, dataset splits) hinder replication.

A2. We appreciate your feedback on ensuring reproducibility. Below we clarify the missing details and emphasize the existing descriptions in our paper:

• Random Seeds. Ours experiments use a fixed random seed of 42 for PyTorch/Numpy initialization.
• Dataset Splits for AdaptEval Benchmark. The proposed AdaptEval benchmark is specifically designed for TTL. Unlike traditional benchmarks, AdaptEval contains only test-domain data without predefined train/val/test splits, as TTL methods adapt solely during inference. Full dataset construction protocols are provided in Supp. B.
• Additional Reproducibility Guarantees. 1) All hyperparameters (learning rates, batch sizes, adaptation steps) are detailed in Sec. 5.1. 2) Hardware specifications (e.g., GPU types) and library versions (PyTorch) are detailed in Supp. C.2. 3) Code, pre-trained models, and AdaptEval data will be publicly released upon acceptance.

Q3. Consider moving supplementary metrics (e.g., BERTScore) to the main paper for a richer evaluation.

A3. We appreciate the feedback and agree that incorporating supplementary metrics like BERTScore into the main paper will strengthen the evaluation's comprehensiveness.

Q4. Expand the discussion of trade-offs in the online setting (e.g., adaptation quality vs. efficiency).

A4. Thanks for your valuable comments. We conduct additional experiments in the online setting to evaluate the effect of batch size under streaming-like scenarios (see Table C). With bs=1, the model updates after each test sample, simulating a true streaming setting. While this increases update steps, our method remains effective. Using bs=100 reduces update frequency while slightly improving performance. This indicates that accumulating more samples before updating provides more reliable learning signals, and thus enhances the adaptation performance. The results suggest that in an online setting, adopting as high a batch size as possible can lead to both effective and efficient performance. The results suggest that in the online scenario, while more frequent updates offer immediate adaptation, adopting a larger batch size can lead to more effective and efficient adaptation.

Table C. Experimental results in the Online setting on DomainBench.

Q5. I did not quite understand the Eq 4, why Minimizing the perplexity to the input $\mathcal{P}(x;\Theta)$ is equivalent to maximizing the input generation probability $P(x;\Theta)$.

A5. Thank you for your insightful suggestion. The equivalence between minimizing input perplexity and maximizing the input generation probability is a fundamental property of perplexity in probabilistic models. We clarify this relationship as follows. Perplexity $\mathcal{P}(x;\Theta)$ for a sequence $x$ is defined as the exponential of the cross-entropy loss $P(x;\Theta)$, Formally, Eq. (2) can be written as：$\mathcal{P}(x;\Theta)=e^{-\frac{1}{N}\log P(x;\Theta)}$, where $N$ is the number of tokens in the sequence. Therefore, maximizing $\log P(x;\Theta)$ directly corresponds to minimizing $\mathcal{P}(x;\Theta)$.

---

We sincerely hope our clarifications above have addressed your concerns.

We thank the reviewer for the encouraging comments and detailed suggestions. Responses are below:

Q1. Regarding the claim of "Reducing Output Perplexity through Input Perplexity Minimization," the theoretical justification provided is somewhat unclear.

A1. We appreciate your feedback. We clarify our motivation below:

• Theoretical Motivation Clarification. In our study, we observe the strong positive correlation between input and output PPL (see Fig.1b). Relying on this, we seek to reduce the output PPL by input PPL minimization.
• Autoregressive Training Dynamics. In autoregressive models, each token is generated using the internal representation of the input. Although the model parameters are updated during input PPL minimization, this process refines the representation $h(x;\Theta)$, which in turn informs the generation of $P(y|x;\Theta)$. In other words, since $P(y|x; \Theta)$ is generated directly from $h(x;\Theta)$, an improved representation of $x$ can lead to more accurate and confident next-token predictions, which is expected to reduce output PPL.
• Empirical Validation. We compare the changes in both input and output PPL between our proposed method and the original LLM. In Tab. A, our method not only reduces input PPL but also leads to a consistent decrease in output PPL.

Tab. A: Experimental results (metric is PPL) on DomainBench.

Q2. Need to empirically test various values of $P_0$ on reasoning-oriented tasks. Introducing a more generalizable or theoretically justified criterion for selecting $P_0$ would further strengthen the method.

A2. We would like to clarify the applicability of the threshold $\mathcal{P}_0$ in reasoning-oriented tasks, and outline potential strategies for its dynamic selection:

• Generalization of $\mathcal{P}_0=e^3$ in reasoning-oriented tasks. We conduct experiments with values of $\mathcal{P}_0=\\{e^2,e^3,e^4\\}$. In Tab. B, when $\mathcal{P}_0=e^3$, our method achieves the best performance on three datasets. These results indicate that $\mathcal{P}_0=e^3$ generalizes well to reasoning-oriented tasks.
• Potential method for dynamic threshold selection. To address the concern regarding the generalizability of the static threshold, one can adopt a dynamic threshold selection scheme. Instead of a fixed $\mathcal{P}_0$, we define $\mathcal{P}_0^{(t)}=\mu\_{\mathcal{P}}^{(t)} + \alpha \cdot \sigma\_{\mathcal{P}}^{(t)}$, where $\mu\_{\mathcal{P}}^{(t)}$ and $\sigma\_{\mathcal{P}}^{(t)}$ are the mean and standard deviation of test samples PPL at step $t$. This method automatically identifies high-surprisal samples critical for adaptation while excluding redundant low-PPL data.

Tab. B: Effects of different $\mathcal{P}_0$ under Llama3.1-8B on ReasoningBench.

Q3. There is a need to evaluate newer and larger-scale models from multiple model families.

A3. We strengthen our empirical validation by expanding experiments across model families and scales. Specifically,

• Existing Multi-Family Model Coverage. As shown in Tables 2-3, our experiments already encompass: 1) Llama Family: 3B, 8B, and 13B parameter variants. 2) Qwen Family: Latest Qwen2.5-7B-Instruct model.
• New Model Family Added. To further validate broader applicability, we conduct additional experiments with Phi-4-14B. In Tab. C, our method consistently outperforms existing baselines, demonstrating its effectiveness on larger-scale model adaptation tasks.

Tab. C: Experimental results on DomainBench.

Q4. Need to add SOTA TTA methods for a more comprehensive evaluation.

A4. We agree that evaluating against more recent TTA methods can provide a more comprehensive comparison and strengthen the empirical validation of our method. We include COME [r1], a recently proposed TTA method accepted at ICLR 2025, in our experimental comparison. In Tab. D, the results demonstrate that our method consistently and significantly outperforms COME across all domains.

Tab. D: Comparison between COME and our method on DomainBench.

[r1] COME: Test-time Adaption by Conservatively Minimizing Entropy, ICLR 2025.

---

We sincerely hope our clarifications above have addressed your concerns. We would be grateful if you could kindly reconsider the evaluation of our paper.

We sincerely appreciate your high level of encouragement for our work. Your recognition of "provides a solid rationale for the necessity of TLM as a solution", "the paper strongly justifies the efficacy of perplexity-based optimization", and "AdaptEval appears likely to serve as a unified standard for subsequent research" is deeply appreciated. Our detailed responses are below:

Q1. Although the proposed method achieves generally positive experimental results, there are no concrete examples of actual outputs included (only input/output examples of the datasets are mentioned), making it less clear how the model performs in real usage scenarios.

A1. We sincerely thank you for highlighting the need for concrete examples to demonstrate practical efficacy. As shown in https://anonymous.4open.science/r/ICML1861, our method consistently outperforms SOTA methods across diverse scenarios. Our analysis is detailed below:

• More accurate reasoning steps. In reasoning tasks, our method produces logically sound and well-structured intermediate steps, enabling the model to reach correct conclusions more reliably.
• Better coherence and fluency. Compared to TTA methods, our method maintains output fluency and avoids the repetition issues commonly observed in models like Tent or EATA.

Q2. Recent TTA methods not discussed. Moreover, a direct comparison to fine-tuning and RAG approaches is missing, which is regrettable.

A2. According to your suggestions, we further compare with a SOTA TTA method, COME (ICLR'25,[r1]). From the results in Table A, our method consistently and significantly outperforms COME across all domains. This further demonstrate the superior performance of our method in addressing LLMs adaptation to new domains using only test data.

Experimental Design Rationale with RAG and fine-tuning approaches. Our experiments focus on TTL, which uses only unlabeled test data without relying on additional labels or retrieval resources. Fine-tuning typically depends on task-specific labeled data, while RAG assumes a well-maintained knowledge base that may not generalize across domains. In contrast, our method updates solely on unlabeled test data via perplexity minimization. Therefore, we compare with TTA methods that share similarly label-free, resource-free settings.

Table A: Comparison between COME and our method on DomainBench.

[r1] COME: Test-time Adaption by Conservatively Minimizing Entropy, ICLR 2025.

Q3&Q4. More detailed comparisons of time complexity and resource requirements would be welcome when handling different types of domains (e.g., conversational tasks, online streaming data).

A3&A4. Thank you for the valuable comments. According to your suggestions, we add experiments under the online streaming setting (batch size = 1), which more accurately reflect real-world inference conditions, such as conversational or sequential inputs. Detailed analyses are as follows:

• Time Complexity. In the TTL setting, total computation time $T_{\text{total}}$ consists of forward passes plus backward updates, i.e., $T_{\text{total}} = T_{\text{forward}} + T_{\text{backward}} \cdot N_{\text{selected}}$, where $N_{\text{selected}}$ denotes the number of samples used for backpropagation. Thanks to our Sample Efficient Learning Strategy, our method significantly reduces $N_{\text{selected}}$ compared to baselines. For instance, as shown in Table B, our method requires up to 45% fewer backpropagation than EATA, resulting in lower overall computation time.
• Resource Requirements. As shown in Table B, we adopt LoRA for parameter updates, thereby limiting the trainable parameters to a small fraction of the full model. This design keeps memory usage and computational overhead low, especially when compared to full-parameter tuning. Consequently, our method enables practical adaptation in resource-constrained environments.

Table B. Experimental results of our proposed method in the Online setting (batch size=1).

---

We sincerely hope our clarifications above have addressed your concerns. We would be grateful if you could kindly reconsider the evaluation of our paper.

Q1. The claim "This improved representation ..., thereby reducing conditional output PPL" is not properly justified either theoretically or empirically.

A1. We thank the reviewer for this important observation. We agree that the original statement regarding conditional output PPL reduction requires more precise formulation and supporting evidence. Please allow us to clarify:

• Causal Relationship Clarification. We have revised the phrase "thereby reducing conditional output perplexity" to "This improved representation facilitates more accurate and confident next-token predictions, which is expected to reduce output perplexity".

• Theoretical Motivation Clarification. In our study, we observe a strong positive correlation between input and output PPL (see Fig.1b). Relying on this, we seek to reduce the output PPL by input PPL minimization.

• Empirical Validation. We compare changes in both input and output PPL between our method and the original LLM in Tab. A. Our method reduces input PPL and consistently decreases output PPL.

Tab. A: PPL results on DomainBench.

Q2. Add PPL as metric.

A2. As suggested, we include PPL as a metric in Tab. B. Our method outperforms existing methods in terms of PPL.

Tab. B: PPL results on DomainBench.

Q3. The training/restoration setting is not clear.

A3. In our method, the model weights are not restored to their original state after processing each test sample. Specifically,

• Offline Setting. All test data are processed at once, and the model is updated using all available test samples before any testing.
• Online Setting. Test samples are processed sequentially, with the model updated after each test sample or mini-batch, and the model parameters are never reset.

Q4. Differences of the proposed method from [R Sennrich].

A4. Both works use PPL minimization, and our method differs from [R Sennrich] as:

• Problem Setting. We focus on adapting LLMs at test time using unlabeled test data to handle distribution shifts. In contrast, [R Sennrich] assumes access to labeled source-domain training data and optimizes translation models from a static source corpus.
• Optimization Objectives. We adopt input PPL minimization as the optimization objective, enabling efficient self-adaptation of LLMs to target domains during testing. Instead, [R Sennrich] optimizes output PPL using labeled data.

Q5. Justification of batch setting needed.

A5. We evaluate batch size in the online setting (bs=1 vs 100) under streaming-like scenarios. With bs=1, the model updates after each test sample. Results demonstrate increased update steps（1514→2541), while our TTL remains effective (0.2040→0.1917).

Q6&Q7. Differences between TTA and TTL? Also, PPL and entropy?

A6&A7. Both TTA and TTL adapt models without labeled data, but they differ in key aspects:

• Objective Function. Most TTA methods minimize the output entropy $H(P(y|x))$. TTL minimize the PPL of the input sequence $\mathcal{P}(x)=e^{1/T\sum_{t=1}^TH_{CE}^{(t)}}$, where $H_{CE}^{(t)}$ is the token-level cross-entropy.
• Task-Specific Adaptation. TTA seeks to improve output confidence but cannot optimize the input representation quality that is essential for LLMs. In contrast, TTL minimizes input PPL to refine the internal representation of the input sequence to encourage more accurate predictions.

Q8. Can you show a comparison of Fig.1b, which presents an updated input and output PPL after input PPL minimization?

A8. Fig. 1(b) shows a strong positive correlation between input and output PPL on DomainBench using Llama3.1-8B with varying degrees of training. We compare the changes in both input and output PPL of our method with the original LLM in Tab. A, our method reduces input PPL and consistently decreases output PPL.

Q9. Fig.1c meanings and Training Protocol.

A9. Clarifications are below:

• Fig. 1c shows how the selection of different proportions of test samples (X-axis: top/bottom p% by PPL) would impact the final performance for TTL (Y-axis: evaluated on the full test set).
• Training Protocol. We pre-sort test samples by PPL, select a given proportion, randomly shuffle the subset, and perform TTL. The adapted model is then evaluated on all test data.

---

We sincerely hope our clarifications above have addressed your concerns. We would be grateful if you could kindly reconsider the evaluation of our paper.