The Surprising Effectiveness of Test-Time Training for Few-Shot Learning

Thank you for the positive review! We are happy to answer any new questions you have later during the rebuttal process.

Thank you for your review and the feedback on our paper.

Q1. Definition and Application of Test-Time Training (TTT)

We use the definition of TTT provided by (Sun et al. (2020)): self-supervised training a model using the unlabeled test sample $d_{\textrm{input}}$. Our work applies this definition in the few-shot learning setting where $d_{\textrm{input}}$ consists of instance-specific few-shot demonstrations and a query. The core idea remains: adapting model parameters at inference time based on information available for the specific test input being processed.

Finally, LoRA is simply the means by which we implement TTT in this paper. it is a simple and efficient fine-tuning method that enables us to do TTT with a 8B-size model. In practice, our TTT framework can be used with other fine-tuning techniques as well.

We can clarify more if the reviewer has more questions.

Q2. Most of the improvements come from inference strategies

We kindly disagree. Our experiments on the Big-Bench Hard (BBH) show that the TTT without any task-specific voting mechanisms or data augmentations still yields significant improvements. Moreover, our vanilla method (Fig. 7), which excludes any augmentations or voting procedures, still achieves a substantial performance boost—solving 22 tasks compared to just 5 (no TTT), marking a 440% improvement.

Q3. Can you clarify the contributions and significance?

Thank you for asking for clarification, and we will update the text to reflect our contributions.

We presented a successful extension of the TTT paradigm to the few-shot learning setting and did a systematic study of design choices for this setting. To the best of our knowledge, we are the first to apply TTT within the in-context learning framework.

We showed substantial improvements (e.g., up to 6x on ARC, 7.3pp on BBH) over strong baselines on 2 popular benchmarks. The magnitude of these improvements highlight the limitations of standard ICL and opens up a new avenue for LM research with TTT.

Q4. TTT Dataset Statistics

The size of the dynamically generated $D_\textrm{TTT}$ depends on the number of demonstrations (K) and the number of augmentations/permutations used.

For ARC, we use $K$ demonstrations (typically 2-7), generate $K$ leave-one-out tasks, apply $|T|$ geometric transformations (Section C.1 lists 20 base transformations/compositions), and n=2 permutations (line 1066), leading to roughly $K * |T| * n$ synthetic examples per task, capped at 250 (line 202). For BBH, we use $K=10$ demonstrations and $n=40$ permutations (line 344), resulting in $10 * 40 = 400$ leave-one-out examples per task.

We will add these specific calculations/details to Appendix C.2 (for ARC) and F.1 (for BBH) to clarify the scale of the temporary training data used for TTT.

Q1. Have authors tried domains such as coding or math? This paper produced a SoTA way to apply TTT to LMs in the few-shot learning setting using the challenging ARC-AGI and BB-Hard as our reasoning problem sets. Note that our implementations do not leverage the CoT capabilities of the models. We believe extending this to non-few-shot learning settings and to domains where CoT is crucial is a very exciting research problem and we’re currently exploring these extensions.

Q2. Recent findings show RL-trained models generalize better. Can TTT still be advantageous if the base model is strongly RL-trained? That is a great question! Recently, there has been a lot of interest in using RL to unlock long reasoning abilities in LMs. As mentioned in our previous answer, combining TTT with CoT models (both RL or no RL) is a promising future direction! Similarly, another possible extension is Test-Time-RL, where RL is incorporated into the test-time training process (Simonds and Yoshiyama, 2025).

Q3. Many benefits disappear when actual scaling happens, how will this affect TTT? In Section 4.4, we present scaling results for Llama models of 1B, 3B, and 8B parameters, where TTT improves performance by 480%, 163% and 157% respectively. Thus, TTT scales effectively with increasing model size.

Q4. Augmentations makes the method a little complicated Our experiments on the Big-Bench Hard (BBH) suite of tasks show that the core method of fewshot TTT without any task-specific voting mechanisms or data augmentations still yields significant improvements. Moreover, our vanilla method (Fig. 7), which excludes any augmentations or voting procedures, still achieves a substantial performance boost—solving 22 tasks compared to just 5 (no TTT), marking a 440% improvement.