Predicting the Performance of Foundation Models via Agreement-on-the-line

Dear Reviewer,

Thank you very much for your time and thorough feedback! We have now made the following changes in response to your feedback:

1. To illustrate the significance of the random head initialization for obtaining ensembles exhibiting Agreement-on-the-line (AGL)
   • We study CLIP full fine-tuning with CIFAR10 on CIFAR10-C Shifts (see Table 7)
   • We extend the results of CLIP linear probe fine-tuning to the Office-Home dataset (see Table 6)
   • We train OPT and BERT models with linear probe fine-tuning for Question Answering (see Table 8 and 9)
   • We add a new Text Classification study (with both linear probe, and full fine-tuning) using GPT, OPT, and LLaMa (see Table 10)
2. We have changed the metric from MAE (Mean Absolute Error) to MAPE (Mean Absolute Percentage Error) to show the significance of the OOD accuracy estimates, as the MAPE captures the relative not just absolute deviation from the true value. We have also added a table with the MAE for different OOD accuracy ranges (Appendix Section 9.12) to show the consistent performance of ALine across high- and low-accuracy models.
3. We have provided some preliminary intuition as to the specific advantage of random linear head initialization for obtaining ensembles which exhibit AGL in the response below. We can add this in our paper if it would be helpful. We address your concerns below in more detail.

[Use of the word diversity] I think the authors use the word diversity without explaining too much what exactly they mean and what is needed in order to make the agreement-on-the-line method work. Clearly there needs to be some diversity (otherwise all models would agree everywhere as correctly pointed out by the authors) but can there be too much diversity?

Thank you for the feedback! We apologize for the confusion caused by our overloaded use of the term diversity.

1. At a high level, like you rightly said, we need some diversity in model predictions, else the models will agree too much. However, not just any kind of diversity is sufficient—for example, just adding noise to the models’ predictions would make them diverse, but not in a way that would help us use agreement to predict OOD accuracy. It is not just an overall measure of diversity that matters, but the “type” or “form” of diversity.
2. This paper shows that random head initialization leads to the right kind of diversity in the predictions of lightly fine-tuned models; thus causing the agreement and accuracy lines to have the same slope and intercept i.e. Agreement-on-the-line (AGL). Other sources of diversity lead to just ACL (i.e. strong linear correlation between the ID and OOD accuracies).
3. We will make this clear in our revised draft, that the big challenge in using foundation models is introducing some diversity, as just having any diverse ensemble on its own is not sufficient.
4. If we use models that are trained on drastically different training sets, they might be “too diverse”, in that the accuracies of these models might follow completely different linear trends between ID and OOD accuracy (i.e. ACL itself might not hold). We clarify again that we do not claim to have a precise measure of diversity, and even more so, AGL requires the right type of diversity.

[Conceptual understanding] Why does only using differently initialized random heads give the “right” amount of diversity?

This is indeed a surprising observation, and we think it would be interesting future work to understand this precisely. In particular, we find this observation to be true even for linear probing on CLIP features, which is potentially amenable to theoretical analysis. Our (admittedly imprecise) intuition is that since we are doing only a few steps of fine-tuning, we never move too far from the random head initialization. Hence with different initializations, we end up with models that can be quite “far” from each other, despite starting from the same foundation model. This paper [3] shows that random initialization dictates how the pretrained features end up changing, and provides some insight into why the head initialization can be so important. On the other hand, if we fix the random initialization, we might not end up moving too far, and as a result even with different data subsetting or data ordering, we might end up with very similar models that agree a lot ID and OOD, i.e. causing the agreement line to lie much higher than the accuracy line.

Dear Reviewer,

Thank you for your time and feedback! We have now made the following changes in response to your feedback:

1. We have added a paragraph in the introduction summarizing our main contributions and what insights we add compared to prior works
2. We have created a new Section 5 discussing the comparison between the ALine method and the other baseline methods.
3. We have made the table and image captions more descriptive.
4. We have expanded our study of CLIP for Image Classification to full fine-tuning, and shown that the random linear head initialization is still the only method consistently yielding ensembles exhibiting Agreement-on-the-line (see Table 7).
5. We have provided some preliminary intuition as to the specific advantage of random linear head initialization for obtaining ensembles which exhibit AGL in the response below. We could add it to the paper if it’s helpful.

We address your concerns below in more detail:

 

Novelty and new perspectives

Thank you for your feedback! We agree that the main “novelty” in our study is that we study a new setting of fine-tuning foundation models. Methodologically, we do borrow heavily from prior work of Agreement-on-the-line [1], but we make some important changes to make it work in this setting. Prior work uses an ensemble of independently trained models, but this is intractable in the new setting. Therefore, we study three different ensembles that are tractable, and find that random head initialization is the only ensemble that works, but it works surprisingly well! We think our observations add the following new insights into the broader phenomenon of “agreement-on-the-line”, some of which surprisingly contradict hypotheses or suggestions made in prior works.

1. No prior work shows any distinction between the agreements afforded by different ways of generating ensembles—for example data subsetting, or different random initializations etc. We find that random head initialization is therefore a special method of introducing diversity. We hope our work inspires future theoretical studies on the effect of random head initialization.
2. [1] found that linear models when trained from scratch do not show AGL. However, we see that linear models on top of CLIP features show AGL, suggesting that it’s less a property of the model family and more a property of the data distribution. This observation also makes it more tractable to study AGL theoretically since linear models are easier to analyze than neural nets
3. Prior work in ACL [2, 5] suggested that different pre-training data for vision models would lead to different effective robustness i.e. the trend between ID and OOD performance. We find the contrary in Section 4, that different pretrained language models can observe the same robustness.

 

Intuition for why random head initialization is different

Our (admittedly imprecise) intuition is that since we are doing only a few steps of fine-tuning, we never move too far from the random head initialization. Hence with different initializations, we end up with models that can be quite “far” from each other, despite starting from the same foundation model. This paper [4] shows that random initialization dictates how the pretrained features end up changing, and provides some insight into why the head initialization can be so important. On the other hand, if we fix the random initialization, we might not end up moving too far, and as a result even with different data subsetting or data ordering, we might end up with very similar models that agree a lot (agreement line is much higher than accuracy line). If this explanation is helpful, we’d be happy to add it to our paper or discuss more.

 

Some details are either deferred to the appendix (definition of ALINE) or the reader is directed to prior work (methods that utilize model confidence in Section 4.2), which makes it harder to follow some details.

Thank you for the feedback! We have now added a sentence to summarize what ALine is doing in our baseline comparison section (Section 5).

Additional changes

1. (Minor) We have removed hyperparameters from the four sources of diversity as the category is too generic for a controlled study. This is reflected in the paper as well. Thus, we have three sources of diversity: random head initialization, data shuffling, and data subsetting. We’d also like to emphasize that this does not change our results – random head initialization induces AGL most effectively.
2. We have changed the error metric of ALine in the main body from mean absolute error (MAE) to mean absolute percentage error (MAPE) as the MAPE captures the relative not just absolute deviation from the true value. However, we have also included the MAE by accuracy range in the Appendix (Section 9.12).

[References]

[1] Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. “Agreement-on-the-line: Predicting the performance of neural networks under distribution shift.” 2022.
[2] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. “Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization.” 2021.
[3] Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter. “Assessing generalization of sgd via disagreement. International Conference on Learning Representations” 2022.
[4] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. "Fine-tuning can distort pretrained features and underperform out-of-distribution." 2022.
[5] John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. “The effect of natural distribution shift on question answering models.” 2020.

Figures captions could also use more details and be more self-contained (e.g. Figure 3, Figure 4 -- which column is what method?).

Thanks for pointing this out, and apologies for the unwanted confusion. We have rewritten all of our table and image captions to briefly recap the experiment, the dataset and methodology, and the drawn conclusion. Specifically,

1. In Figure 3, we have updated our caption to - ACL and AGL observed on extractive Question Answering when computed over a set of models fine-tuned from different base foundation models. ACL and AGL are seen to hold for all four shifts of SQuAD-Shifts under this setup. The base models used here are OPT, GPT, and LLama .
2. Figure 4 has been moved to Figure 6 in the updated paper with the caption - ID vs OOD trends of accuracy and agreement of LLMs fine-tuned for Question Answering from a single pretrained base model (GPT2-Medium). Each column presents trends for different sources of stochasticity employed to obtain a diverse ensemble of fine-tuned models. We see that random linear head initialization is the best method to obtain a set of models exhibiting AGL behavior.

 

Questions:

In Figure 4, which column corresponds to which method? Visually it seems like the second column is the best, and for no column does it seem like AGL always holds.

We apologize for the confusion. We have added labels for Figure 4 (the new Figure 6), it is random head initialization, data shuffling, and data subsetting from left to right. AGL holds the best with random head initialization, and the respective ALine (a method which leverages this proximity of the lines for estimating accuracy) MAPE errors can be found in table 2, which reaffirms this.

In Section 4.1 the phrase “ensemble of foundation models” is a bit confusing. Does it mean an ensemble in the machine learning sense or just a "group" of models?

We apologize again for the confusion, we use ensemble here simply to refer to a collection of models. This usage is consistent with [1] and [3].

In Section 4.2, how is temperature scaling relevant in this setting?

We temperature scale all our baselines which are confidence based, and report the numbers with the best performance between with and without temperature scaling.

What is $\Phi$ in Section 8.3?

We apologize for the misclarification. $\Phi^{-1}$ refers to probit scaling which is applied to induce a better linear fit as in [1] and [2]. We have added the explanation in the Appendix (Section 9.2).

Is there a specific reason to only do linear probing for the CLIP setting, and just 1 fine-tuning task of SQuAD for the LM setting?

Thanks for your question. We initially picked linear probing for CLIP since we wanted to consider the “lightest” form of fine-tuning. However, we have now added full fine-tuning as well.

1. See our full fine-tuning experiments for CLIP based Image classification on the CIFAR10 dataset in Appendix Section 9.3.
2. For SQuAD in the LM setting, linear probing resulted in poorly performing models and was hence omitted.
3. We have included Text classification in Appendix Section 9.6/9.7 where we perform both full-finetuning and linear probing. Overall, our finding of the need for random head initialization for AGL, holds across all modalities, datasets, and fine-tuning methods.

We present a summary of our experiments below:

Image Classification with CLIP

• Full Fine-tuning:
  1. (Newly added) CIFAR10 Image classification: ID dataset: CIFAR10; OOD datasets: CIFAR10-C
• Linear Probing:
  1. CIFAR10 Image classification: ID dataset: CIFAR10; OOD datasets: CIFAR10-C, CIFAR10.1
  2. (Newly added) OfficeHome: We use the standard domain adaptation benchmark with 4 domains: “Art”, “ClipArt”, “Product”, “Real”. We test all 3x3 settings where each domain is considered ID one at a time, with the remaining 3 domains being OOD
  3. ImageNet: ID dataset: ImageNet, OOD datasets: ImageNetV2, ImageNetC
  4. Satellite imagery: fMoW-WILDS
  5. Species classification from camera traps: iWildCam-WILDS
  6. Pathology: ID dataset: Camelyon17-WILDS

NLP tasks

• Full Fine-tuning:
  1. Question answering: ID dataset: SQuAD; OOD datasets: (i) SQuAD-Shifts - Reddit (ii) SQuAD-Shifts - Amazon (iii) SQuAD-Shifts - NYT (iv) SQuAD-Shifts - New Wiki
     • Single Base Model: GPT2-Medium, OPT-125M (Newly added), BERT (Newly added)
     • Multiple Base Models: GPT2 variant, GPT-Neo, OPT variants and Llama2
  2. (Newly added) Text classification: ID dataset: MNLI-matched, OOD datasets: (i) MNLI-mismatched (ii) SNLI (iii) WNLI (iv) HANS
• Linear Probing:
  1. (Newly added) Text classification: ID dataset: MNLI-matched, OOD datasets: SNLI

Dear Reviewer

Thank you very much for your time and helpful feedback! We have now made the following changes:

1. We have clarified the role of different sections in the response below.
2. Added a new baseline method, i.e. ProjNorm [5] (Appendix, Section 9.10), for OOD performance estimation for the Question-Answering model ensembles.
3. We have provided some preliminary intuition as to the specific advantage of random linear head initialization for obtaining ensembles which exhibit AGL in the response below. We could add it to the paper if it’s helpful. .

Paper Organization (Putting Section 3 with more contributions after Section 4)

Thank you for your feedback! We apologize for any confusion about the ordering of our contributions.

1. We’d like to clarify our reasoning for placing Section 3 before Section 4. Our primary focus was studying Agreement-on-the-line (AGL) in the setting wherein we fine-tune models from a single base foundation model. There may be many situations where we do not have access to multiple foundation models.
2. Furthermore, since several foundation models are “black-box” in different ways (we don’t always know what they are trained on, for how long, what architecture etc), there’s always the risk of models being too different to have their fine-tuned counterparts lie on the same ID-OOD accuracy line. Thus Section 4 was just to show an interesting independent and complementary observation that in some scenarios, where we do have access to multiple foundation models, AGL seems to hold when using this ensemble of existing models.
3. It is true that Section 3 requires us to be smart when creating the ensemble for evaluating AGL, while Section 4 allows us to use existing models directly. But we think Section 3 is the more general setting that’s of primary interest. We can make this more clear in our discussion of different sections and layout of the paper. Does this address your concern or do you still think we should reorder our sections?

Limited novelty

1. We agree that in terms of methodological advancements our novelty might seem limited. However, we strongly believe that our paper makes evaluative/conceptual advancements to understand the phenomenon of agreement-on-the-line (AGL) in the setting of fine-tuning foundation models, where we do not have access to independently trained models from scratch.
2. As the reviewer notes, AGL is not a universal phenomenon, and we need to identify the right ensemble of models to use. Prior work takes a set of independently trained models from scratch, trained on the same dataset. This is not feasible in the setting of foundation models—we simply cannot train multiple foundation models on the same dataset. In fact, apriori we did not expect any form of AGL to hold with fine-tuned models from the same foundation model, because we expected their error to be highly correlated.
3. We perform a systematic evaluation of different ways to create ensembles of fine-tuned models, and find that only one method worked reliably across the board: having different initializations of the random head. We believe this is a finding of great practical relevance because it allows us to perform unsupervised accuracy estimation when using foundation models. Our method achieves state-of-the-art.
4. We also believe this work advances our understanding of the broader phenomenon of AGL. For example, [4] found that linear models when trained from scratch do not show AGL. However, we see that linear models on top of CLIP features show AGL, suggesting that it’s less a property of the model family and more a property of the data distribution. Prior work in ACL [3] suggested that different pre-training data for vision models would lead to different effective robustness i.e. the trend between ID and OOD performance. We find the contrary in Section 4, that different pretrained language models can observe the same robustness.

We hope that this discussion also clarifies the novelty and scientific value of our work. To summarize, we provide a rigorous study of limitations of applying AGL for foundation models. Methodologically, this also lends us SoTA estimation of the performance of foundation models. We have added a contributions section in the introduction to clarify the novelty and can further improve upon our writing for the camera-ready version. Thank you again for your time, please let us know if there is anything else you would suggest to help clarify our work.

Thank you again for the comprehensive and useful review! As you've suggested, we had incorporated more baselines (i.e. ProjNorm) and clarified our contributions (our work, in addition to providing state-of-the-art performance estimation of large foundation models, debunks several hypotheses about agreement-on-the-line and accuracy-on-the-line from previous studies). We've also provided some intuition for why random linear heads may be important to observe agreement-on-the-line, although we leave a more rigorous theoretical analysis for future work. We hope this addresses your concerns.

Do you have any further questions? If you have any more suggestions or feedback, please let us know.

Comparative Analysis: …no benchmarking against existing Out-Of-Distribution (OOD) performance estimation methods…

We apologize for any confusion about the lack of baselines. Our submission does compare to other baselines (Naive Agreement, ATC, AC, DF) in Table 3. We apologize for the table being in Section 4.2, which would have been confusing. The table reports numbers we have calculated over both models with randomly initialized heads (Section 3) and models from different bases (Section 4).

In summary, ATC [1], AC [2], and DF [3] utilize model confidence to estimate OOD accuracy while Naive Agreement [9, 10] directly utilizes agreement to predict accuracy. We have also added a new baseline of ProjNorm [4] in Section 9.10 which yields a score correlated with the OOD performance of the model. We see that our proposed method significantly outperforms baselines. The table below is a snapshot of some of our baseline comparisons; a full picture can be found in Table 3. Based on your feedback, we have moved our baseline comparison section to its own standalone Section 5, and provided more details about the evaluated datasets, models, and baselines. Please let us know if there are any other baselines we should add.

Data Diversity … The volume and variety of datasets used for validation appear limited

We list below the different datasets we evaluate on, including some that we added based upon your feedback (let us know if you feel this helps to address your concerns, and if there are particular new datasets that would be valuable to add in your opinion).

Image Classification with CLIP

1. CIFAR10 Image Classification: ID dataset: CIFAR10; OOD datasets: CIFAR10-C (which includes 19 different shifts; we test on each individually), CIFAR10.1
2. (Newly added) OfficeHome: We use the standard domain adaptation benchmark with 4 domains: “Art”, “ClipArt”, “Product”, “Real”. We test all 4x3 settings where each domain is considered ID one at a time, with the remaining 3 domains being OOD
3. ImageNet: ID dataset: ImageNet, OOD datasets: ImageNetV2, ImageNetC
4. Satellite imagery: ID/OOD dataset: subsets of fMoW (WILDS)
5. Species classification from camera traps: ID/OOD dataset: iWildCam (WILDS)
6. Pathology: ID/OOD dataset: Camelyon-17 (WILDS)

NLP tasks with GPT, Bert, OPT, Llama

1. Question answering: ID dataset is SQuAD; OOD datasets are (i) SQuAD-Shifts - Reddit (ii) SQuAD-Shifts - Amazon (iii) SQuAD-Shifts - NYT (iv) SQuAD-Shifts - New Wiki
2. (Newly added) Text classification: ID dataset is MNLI-matched, OOD datasets are (i) MNLI-mismatched (ii) SNLI (iii) WNLI (iv) HANS

Our findings on when AGL occurs holds quite robustly across all these settings; i.e. random head initialization while fine-tuning is the only method consistently yielding ensembles where AGL holds, thus allowing for accurate OOD estimation with ALine methods.

Dear reviewer,

Thank you very much for your time and thorough feedback! We have now made the following changes which we believe address all your concerns and strengthen the paper

• Added a paragraph in the introduction clearly describing our contributions and relation to prior work and state of the art
• Clearly mention where we compare to baselines (Table 3), and added one more baseline method of ProjNorm [4]
• Added results on OfficeHome dataset (with 4ID and 3 OOD settings for each) for image classification with CLIP, and MNLI to MNLI-Mismatched, SNLI, HANS, WNLI for text classification (See Appendix Sections 9.6, 9.7) . This now makes a total of 34 different distribution shifts for image classification and 8 different shifts in NLP tasks.
• Considered two new architectures (OPT, BERT) in addition to GPT-2 and Llama for NLP tasks (see Table 8 and 9)
• Studied different kinds of fine-tuning: newly added full fine-tuning of CLIP (Figure 5) in addition to linear probing

We elaborate on each of your concerns below:

Specificity of Contributions: …'agreement-on-the-line' enhances or differs from the current state-of-the-art. To improve, the authors should explicitly state the unique advantages and contributions of their method over existing approaches, possibly by providing a direct comparison to highlight the novelty.

Thank you, to clarify our contributions, we modified the Introduction and added a “contribution” paragraph at the end. To summarize:

• We propose a new state-of-the-art method for unsupervised accuracy estimation under distribution shifts when using large pre-trained models (a.k.a. foundation models) that are lightly fine-tuned for specific tasks. Prior works have primarily dealt with the models trained from scratch, where there is no transfer learning or extensive pre-training to account for. Hence, we believe the setting studied is new, and extremely relevant in today’s context. Prior work does not directly apply in this setting.
• Furthermore, our work leveraging Agreement-on-the-line (AGL) for OOD estimation, builds on top of prior work [11]; but extends it in important ways to make it work in this new and important setting. Prior works have primarily dealt with the models trained from scratch, where there is no transfer learning or extensive pre-training to account for. Hence, we believe the setting studied is new, and extremely relevant in today’s context. Admittedly, it is similar in overall methodology—we compute the agreement between pairs in an ensemble of models. However, the key to making AGL work is obtaining the right ensemble. In [11], this ensemble was just a collection of models trained independently from scratch. A likewise (but infeasible) extension to pre-trained models would require numerous such models, all trained from scratch. Our work shows how to side-step this, by systematically identifying a computationally tractable method of obtaining the right ensemble. Specifically, we show that creating an ensemble with randomly initialized final linear heads and then fine-tuning can allow for AGL behavior, and apply downstream methods for unsupervised accuracy estimation, while other similar forms of ensembling (such as data ordering or data subsetting) do not.
• We believe this is a significant finding of practical relevance, but also points to several interesting phenomena underlying AGL that go beyond previous knowledge. Prior work (section 3.4 in [11]) claimed that AGL does not hold for linear models. However, we find the contrary when using pre-trained features (such as CLIP). Furthermore, other prior work [14] suggests that the effective robustness (i.e. the linear fit b/w ID and OOD accuracy) would change depending on the pretraining data. We find that this is not the case for question answering with different pretrained LLMs. Thus we hope our findings can also advance our understanding of the robustness of ML models, particularly those that leverage foundation models.

Thank you again for the comprehensive and useful review! As you've suggested, we had provided experiments on additional benchmarks (e.g. OfficeHome, MNLI), architectures (e.g. BERT, OPT, Llama), and reworded our contributions. We hope this addresses your concerns.

Do you have any further questions? If you have any more suggestions or feedback, please let us know.