Missing Compute-related Ablations: In multi-turn inference settings, it is crucial to provide the number of FLOPs used to arrive at the final answer. Do the authors have any experiments that demonstrate the scaling curve, with FLOPs on the x-axis and performance on the y-axis?

We thank the reviewer for raising a relevant and interesting point. Below, we discuss how our method can be scaled at test time and provide the scaling curves.

Our unsupervised ICL method can be split into two stages.

1. The first is the task adaptation stage. At this stage, we perform the described multi-turn optimization procedure to obtain labels for a set of unlabelled examples. This results in a labeled support set for ICL inference in the second stage.
2. The second stage is actually the test stage, where we make predictions for new incoming test examples. At this test stage, for each test example, we perform ICL inference using examples with labels found in the first stage as in-context examples.

The cost of the first stage is fixed and single-time, as we do it once for a given task and do not need to do any additional turns on the obtained support set. Therefore, our test time compute scaling depends only on the number of ICL shots used to make predictions for each test example during the second testing stage, as a longer context requires more FLOPs.

Figure: https://imgur.com/a/wzS0QFs

In the attached figure, we show how the performance of the unsupervised ICL changes as we scale the test time compute by increasing the number of ICL examples (blue curves). We perform the experiment for Llama-Instruct 8B and 70B and the RTE dataset. We show that test time compute scaling of the 8B model with our method achieves a better compute-performance trade-off than zero-shot inference with a larger 70B model. Further scaling test time compute with the 70B model further improves the performance. In all cases, we outperform the CoT approach both in terms of compute and performance.

We thank the reviewer, and we will include these results in the final version of our manuscript.

Reliance on Task Encoder Approximation: I think the design of the Task Encoder has not been explored properly in the paper. The authors only did not mention why the same pre-trained model + LoRA can be a good approximation for the task encoder approximation tasks. Why is this case? Based on which ablation in the paper? What are the other alternatives? It’s important to note that, as shown in other research, LoRA often tends to underfit a task, which can lead to suboptimal or inconsistent improvements. This raises concerns that the proposed task inference model may not accurately reflect the tasks that the model is intended to predict.

Indeed, the choice of the parametrization for fine-tuning can lead to different results in terms of under- and over-fitting. Note, that our method can be applied to any fine-tuning strategy, the only difference with supervised fine-tuning is that we do not need labels. We chose LoRA as a widely-adopted adaptation method that trades off over- and under-fitting. To further strengthen our experimental evaluation, we conduct additional experiments of fine-tuning via 1) full fine-tuning and 2) using a linear head on top of the fixed model. We find that LoRA results in overall the best performance for both supervised and proposed unsupervised fine-tuning.

We thank the reviewer, and we will include these additional experiments in the final version.

Limited Applicability to Open-Ended Tasks & Lack of Proper Evaluation in Multi-modal Setting:

We kindly refer the reviewer to the general response for the additional experiments on more recent benchmarks: MMMU, MMMU-Pro, VQAv2, and VizWiz for VLMs and MMLU for LLMs. We show that our method can be applied to them and improve upon the zero-shot baseline. For instance, on the MMLU benchmark, unsupervised ICL brings 3% improvements upon the zero-shot and remarkably matches the performance of supervised ICL. Furthermore, on the challenging MMLU-Pro, our unsupervised ICL improves by 6% upon the zero-shot counterpart. We also observe similar trends on the newly added vision tasks. Indeed, our unsupervised ICL brings 2% absolute improvement upon the zero-shot baseline for the frontier GPT-4o mode on the MMMU datasetl. Moreover, on the challenging VizWiz dataset, our unsupervised ICL achieves 5% absolute improvement upon the zero-shot baseline. Overall, these results highlight the strength of the proposed joint inference framework and its applicability to the frontier models and the most recent challenging benchmarks.

In addition, in response to the reviewer’s request, we have also evaluated the LLaVA-Next-Interleave-7B model on MMMU and MMMU-Pro and observed that this model does not exhibit supervised ICL capabilities on the aforementioned benchmarks. In particular, zero-shot inference results in 40.7% and supervised ICL results in 38.4% on the MMMU dataset. On the MMMU-Pro dataset, zero-shot inference results in 22.1% and supervised ICL results in 19.3%. Therefore, we decided to take Qwen2-VL-72B, the most capable open-source model on this benchmark [1] that can be triggered to generate intermediate reasoning before answering the question. The zero-shot inference on the MMMU-Pro Standard dataset resulted in 49.7%, and our unsupervised ICL achieved 50.6%, improving the best result for the open-source model on this benchmark.

[1] https://mmmu-benchmark.github.io/#leaderboard

We thank the reviewer for the detailed and thoughtful feedback. We are glad that the reviewer recognizes our joint inference framework and appreciates the soundness of our idea to perform unsupervised adaptation of foundation models. We are also glad that the reviewer finds our paper well-aligned with the recent trends to scale inference-time compute to improve the predictions of foundation models. Finally, we are happy that the reviewer appraises our experiments as rigorous, demonstrating the significant gains brought by the proposed approaches across various models and tasks. Below, we provide a detailed response to the reviewer’s concerns, and we hope that the provided clarifications, along with the additional experiments, will increase the reviewer’s confidence in our work.

Model Capacity and ICL Abilities: The proposed method assumes that the pre-trained model already possesses few-shot in-context learning (ICL) abilities, which are typically found in large-scale models (those with over 8 billion parameters). What about models that don't have these ICL capabilities? Can they leverage their context more effectively during joint inference? To be more specific, if the authors use the Phi-1.5 model and conduct unsupervised fine-tuning or unsupervised ICL again, what would be the outcome?

We kindly refer the reviewer to the general response for the discussion on the reliance on the ICL capabilities. Similar to methods like CoT [1, 2], which relies on the model’s reasoning capabilities, our method relies on the ICL capabilities to improve upon 0-shot performance.

We also provide additional results for smaller models. We make two observations:

1. Small models can still exhibit ICL capabilities and improve upon 0-shot in some cases. To show that, we conduct extensive experiments on RTE, SST2 and BoolQ datasets using small models (table below). We find that whenever a small model exhibits ICL capabilities meaning that supervised ICL outperforms zero-shot baseline, one can expect improvements with the unsupervised ICL as well. For instance, on the SST2 dataset with the Phi 1.5 model supervised ICL achieves 42.7% relative improvement over zero-shot and the unsupervised ICL achieves 44% improvement. With the 3B parameter model, supervised and unsupervised ICL outperform zero-shot baseline by a large margin consistently across all datasets.

RTE

SST2

BoolQ

2. For the smallest tried Llama-3.2-1B model, our unsupervised FT method significantly improves upon 0-shot when using a larger model to compute the joint inference objective and provide feedback to the small task encoder (table below). First, it is easy to see that even using the same 1B model to compute the joint inference objective leads to substantial gains in the performance (table below). For example, as shown in the experiment of the previous response, supervised ICL on the RTE dataset leads to only 5% improvement upon the zero-shot baseline. In turn, unsupervised fine-tuning leads to 10% improvement, outperforming the supervised ICL (table below). Furthermore, using larger models for computing the joint inference objective further improves the performance of the small Llama-3.2-1B model, closely approaching the supervised fine-tuning upper bound. For instance, on the BoolQ dataset, using the 70B model in the objective results in a remarkable 24% absolute gain over the zero-shot baseline. We note that this is a one-time adaptation cost per task, after which the inference is made with the low-cost small model.

Unsupervised FT for Llama-3.2-1B model

Again, we thank the reviewer for the insightful suggestion to perform the additional analysis, which we will include in future revisions of our manuscript.

[1] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. NeurIPS 2022.

[2] Large Language Models are Zero-Shot Reasoners. Kojima et al. NeurIPS 2022.

We thank the reviewer for the valuable feedback and for acknowledging the novelty of our joint inference framework. We are glad that the reviewer recognizes the solid theoretical foundation for the proposed approaches. We are also happy that the reviewer appreciates the diversity of the provided experimental study that covers various tasks and models, highlighting the effectiveness of the proposed approach. Below, we address the reviewer’s concerns and we hope that the provided clarifications and the additional experiments help to increase the reviewer’s confidence in our work.

The paper overclaims it's contribution and effectiveness of its approach. In abstract, the paper claims that 'our approach, although unsupervised, often performs on par with supervised approaches that use ground truth labels.' However, its an overclaim for the following reasons: (1) In the results shown in the paper, most of the time supervised approach is better than the proposed approach. (2) Supervised approach uses LoRA instead of full model training.

We thank the reviewer for their suggestion to improve the presentation of our contributions. We would like to note that, as Table 1 shows, we find that for 10 out of 13 considered NLP, unsupervised ICL closes at least 85% of the relative gap, i.e., (uICL - ZS) / (supICL - ZS), between zero-shot and supervised ICL. It is also important to note that supervised ICL uses ground truth examples to perform adaptation to a task, thus it stands as an upper bound for our unsupervised ICL. We believe that our results provide a remarkable insight into how close unsupervised method can come to the supervised counterpart. We will edit the abstract in the future revision of our manuscript to reflect found trends more objectively.

Our unsupervised fine-tuning approach can be combined with any fine-tuning strategy. In response to the reviewer’s request to include full model training and to further support usage of LoRA fine-tuning instead of full model training, we provide the ablation of the task encoder’s design and its effect on the performance of the unsupervised fine-tuning approach. We consider the Llama 3.1-8B foundation model and the SST2, RTE and Boolq datasets as illustrative tasks. We consider three types of the task encoder: (i) LoRA parameter efficient fine-tuning of the foundation model as done by default in the paper, (ii) Full fine-tuning of the entire foundation model, (iii) linear model on top of frozen Llama embeddings of the input instance.

First, unsupervised fine-tuning with all three task encoders substantially outperforms the zero-shot performance on each of the considered datasets. For instance, even with the simple linear task encoder, unsupervised fine-tuning improves by 12%, 10% and 12% on the SST2, RTE and Boolq dataset respectively. On the other hand, we can observe that linear task encoder significantly underfits both LoRA and full fine-tuning. In particular, LoRA fine-tuning outperforms the linear task encoder by 3%, 9% and 3% on the SST2, RTE and Boolq datasets respectively.

Furthermore, it can be observed that supervised fine-tuning with full model training severely overfits on the RTE dataset compared to LoRA parameter efficient fine-tuning. In turn, the unsupervised fine-tuning with full model training inherits this problem and performs worse compared to the LoRA fine-tuning. In particular, compared to LoRA supervised fine-tuning, full supervised fine-tuning drops by 7% on the RTE dataset. This, in turn, results in a 2% drop of the performance of full unsupervised fine-tuning compared to LoRA unsupervised fine-tuning.

Overall, the obtained results suggest that the LoRA fine-tuning provides the practical trade-off between parameter efficiency and the performance. We will include these ablations in the future revision of our manuscript.

We thank the reviewer for the valuable feedback and for recognizing the substantial improvements brought by our method over the zero-shot baselines. We are also glad that the reviewer finds our framework compelling and the proposed approaches broadly applicable to both open and closed source models as well as across modalities. Below, we address the reviewer's concerns and we hope that our response helps to increase the reviewer's confidence in our work.

the main weakness of this paper is clarity…

&

Q1: how does your method answer multiple questions with one model call? What is the exact prompt used and how are the in context examples formatted?

We thank the reviewer for the fruitful suggestions to improve the presentation of our joint inference and the descriptions of the proposed approaches. Table C1 provides the template for a single pair of input instance (in blue) and answer (in red). Although both the proposed approaches do not generate answers for multiple questions with a single model call, they optimize the joint inference objective that, intuitively, corresponds to such behavior.

Unsupervised ICL

In response to the reviewer’s request and to further elaborate on the details, we provide the method figure for the unsupervised ICL approach (New-Fig. 1: https://imgur.com/a/fNnoKNs), which will be included in the final manuscript. As New-Fig. 1 shows, first, our method generates answers for each test example using zero-shot prompting. Subsequently, the method enters the multi-turn optimization phase, where, at each turn, for each example in the test dataset, the model is prompted with randomly sampled in-context examples from the dataset (excluding the test example) using labels from the previous iteration. These examples are fed into the model in the left-to-right order along with the current query input instance to generate a refined answer. In other words, we concatenate the templates for all the pairs with a query instance and feed them to the model. This algorithm can also be seen in Algorithm B2 in L955. As shown in Figure 2 in the paper, this method indeed optimizes the joint inference objective.

Unsupervised FT

In the unsupervised fine-tuning method, at each optimization step, first, the task encoder $\tau_\theta$ generates the answers for each input instance independently $y_n \sim \tau_\theta(x_n)$. Then, given predicted $y$s, parameters $\theta$ are optimized to maximize the joint inference objective $p_{\mathrm{FM}}(y_1, \dots, y_N | x_1, \dots, x_N)$, as defined in Eq. (7) and (8). This corresponds to the optimization problem in Eq. 12. Note that while the task-encoder makes predictions independently, after optimization with the joint objective, it approximates the result of the joint inference as shown in Sec. 4.2 and Eq. (9).

We will include the new method figure and significantly improve the clarity and presentation of both methods in the camera ready version of our manuscript. We will remove unnecessary details and notation and make sure that our method is explained in a clear and concise way using the newly made image.

I am not fully convinced that adding iterations helps for the no weight update setting. if the authors added a 1-iteration baseline to their results this would resolve my concern

Our unsupervised ICL method can be viewed as an iterative optimization approach to maximize the joint inference objective. We note, therefore, that the “1-iteration baseline” also corresponds to our proposed unsupervised ICL method run only for 1 optimization iteration (turn), as it requires ICL inference with labels obtained during 0th iteration (zero-shot initialization). In the main paper, we study the performance of our method when increasing the number of iterations, including 1-iteration. Specifically, Fig. 2 shows that, generally, having more iterations results in a higher performance.

Akin to any other optimization method, the convergence speed of our multi-iteration optimization method can vary between different datasets. For example, Fig. 5 shows that for the CIFAR-100 dataset, one step is enough to converge. In such cases, this means that our method is able to converge fast and outperform the zero-shot baseline (the left-most point on the referred plots.)

I thank the authors for addressing my concerns and adding a methods figure. With these new explanations and experiments I will raise my score to a 6. My one comment would be to please put results for the one iteration baseline for all the datasets in the final manuscript. I may have miscommunicated why i wanted this: I don't think the method is less interesting without this iteration step. In my experience with these iterative approaches they have very diminishing returns and 1 iteration is often similar or only slightly worse performance, and I was curious as to whether the authors saw this as well. From my impression of the authors reply, iterations do have diminishing returns, which I think would be very useful to know. If one iteration is just as good, why do more :)

We thank the reviewer for the positive evaluation of our work and for acknowledging the importance of our framework that enables unsupervised adaptation of foundation models, reducing the need for labeled examples that are expensive to obtain in practice. We are glad that the reviewer recognizes the substantial improvements brought by the proposed approaches, even when compared to the corresponding supervised baselines. We are also glad that the reviewer appreciates the diversity of the provided experiments that demonstrate the effectiveness of our framework in both text and vision-language scenarios. Finally, we thank the reviewer for acknowledging the flexibility and robustness of our framework that is applicable to both open-weight and close-weight models. Below, we provide clarifications to the reviewer’s concerns and we hope that our response helps to improve the reviewer’s confidence in our work.

The effectiveness of unsupervised ICL seems to rely on the model’s inherent in-context learning abilities. This may restrict its applicability to models lacking strong ICL capabilities. It is unclear how does the same method work for other models with relative weak capability.

We thank the reviewer for the comment. In general, smaller models are indeed expected to have weaker ICL capabilities resulting in smaller improvements of the supervised ICL compared to the zero-shot baseline. To further strengthen our experimental evaluation, we evaluate performance of using smaller models and conduct extensive experiments on RTE, SST2 and BoolQ datasets (tables below). As expected, we observe smaller improvements of the supervised ICL compared to the zero-shot baseline. For example, on the RTE dataset, supervised ICL for 1B model brings 11% relative improvement upon the corresponding zero-shot baseline, while supervised ICL for 3B and 7B models bring 22% and 29% relative improvements upon the corresponding zero-shot baselines respectively. Nevertheless, we can observe that even much smaller models can exhibit reasonable in-context learning abilities, and our unsupervised ICL is also applicable to them in that scenario. We also kindly refer the reviewer to the general response for the detailed discussion of reliance of our joint inference framework on the in-context learning capabilities of a foundation model.