Project and Probe: Sample-Efficient Adaptation by Interpolating Orthogonal Features

We thank you for your review. We provide answers to individual points below.

One major limitation of this work is the fact that the project and probe processes are considered solely in the linear model regime due to the pre-trained feature extraction. Also, the theoretical analysis was conducted with a linear model. how would it extended to large-scale problems?

There are two key reasons for focusing on the linear setting. One is that the pre-trained feature extractors are quite good, and training linear heads on top of pre-trained features has been shown to be a powerful approach for a wide range of problems. Furthermore, such a pipeline is computationally cheap to run, as the optimization is linear and does not require fine-tuning the whole backbone.The second is that we are considering the low data regime. With only a small number of datapoints from the target domain, a full feature extractor will overfit and not be able to generalize to new test points. As such, the pre-trained feature extraction and linear project and probe head are key to the method, which is why all the analysis is conducted with both of these. We find that our method does work on challenging real-world tasks, such as FMoW satellite images.

The authors only compare with projection-based baselines. I think comparisons with recent general unsupervised domain adaptation methods are needed.

The problem setting we are studying is different from unsupervised domain adaptation. Whereas unlabeled DA uses unlabeled target data, our setting uses a small labeled dataset from the target distribution; we give a full description in Section 3. Because such methods do not effectively use labeled target domain data, we do not directly compare to them for fairness.

Instead, we have added two additional points of comparison that use both a small amount of labeled target domain data like Pro^2 and additionally use unlabeled data: Pagliardini et al. [47] and Lee et al [27], shown below. While both methods perform well, Pro^2 at least matches the performance with any target data size and sometimes even outperforms the other two methods, especially with few datapoints (2) on Waterbirds and more datapoints (32) on CelebA, despite not using any additional unlabeled data, which may be difficult to obtain.

While learning diverse/orthogonal features is novel in the context of domain adaptation. There is an active line of research that explores this idea in the standard supervised learning setting, such as [1-7]. I think these methods should be discussed in the related work.

Thank you for the comment and references. We have added these references and some discussion in the related work section. Prior works have explored learning diverse or orthogonal features in standard supervised learning settings and we show how diversification can lead to more sample-efficient adaptation to distribution shifts.

If you have any remaining concerns, please let us know. If we have addressed your concerns, we kindly ask that you may consider raising your score.

Thank you for your review. In light of our responses to your individual points, please let us know if you have any remaining concerns.

A critical comparison is missing from the experiments: How does the proposed method perform compared to zero-shot transfer learning methods (i.e., no target training data), as cited in related work?

We provide the zero-shot transfer performance using the same pre-trained backbone (ViT-L-16_SWAG_E2E_v1) of the datasets below.

We see that using a small amount of target data, Pro^2 results in significant gains, particularly on OOD distributions, and thus produces consistently strong performance across a range of test distributions.

For reproducibility, it is necessary to include the numerical values of the experimental results in addition to the line charts.

We have added numerical values of the experimental results to the end of the Appendix in the revised pdf.

We hope that our response has addressed all your questions and concerns. We kindly ask you to let us know if you have any remaining concerns, and - if we have answered your questions - to reevaluate your score.

Thank you for your comments. We provide answers to individual points below.

There are approaches that exploits Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) for the adaptation of the head of a pretrained model with success (Shysheya et al. 2022). Can the authors comment on the differences between their method and these methods? While the paper covers the relation with respect to LDA, it does not seem to mention the relation with QDA. Adding those baselines to the empirical evaluation may be beneficial.

Both LDA and QDA operate in the setting without distribution shift, and are typically used to learn Bayes optimal predictors on in-distribution data samples under assumptions on the class conditional distributions being Gaussians. On the contrary, our method extracts linear classifiers for few shot adaptation on out-of-distribution data. This requires extracting more directions than just the optimal classifiers on in-distribution source data, which we accomplish with our “project” step. Nevertheless, in Corollary 9, we show that Pro^2 with dimensionality d = 1 provably recovers the LDA direction in a shifted homoscedastic Gaussian model. Similarly, under heteroscedastic Gaussian model, minimizing cross entropy loss with a quadratic kernel (feature extractor), will recover the QDA classifier for d = 1. We also show that using higher values of d is critical in adapting to higher degrees of distribution shift. In this sense, the projection of Pro^2 can be seen as a generalization of LDA/QDA. We compare to a wide range of state-of-the-art baselines (e.g. DFR, Teney et al., D-BAT, DivDis, etc) and include a comparison between our method and replacing the projection step with PCA in the Appendix.

Regarding the empirical assessment, was a consistent hyper-parameter search strategy employed across all the evaluated baselines, or was it exclusively used for the proposed model?

As stated in Section 6.1, we employ the same hyperparameter search strategy across all the evaluated baselines. For all comparisons, we tune hyperparameters over 3 different learning rates (0.1, 0.01, and 0.001) as well as 3 different L2 regularization weights (0.1, 0.01, 0.001). We also sweep over 6 different projection dimensions (d = 1, 4, 16, 64, 256, 1024) and report results over 10 runs.

The authors briefly mention the relation with the previous work of Morwani et al. (2023) in the related work section. This section appears somewhat limited and would benefit from a more in-depth exploration. Could the authors elaborate on the parallels and distinctions between their work and Morwani et al. (2023)?

We apologize for any confusion. Morwani et al (2023) is a related concurrent work that studies the simplicity bias of 1-hidden layer networks. Their work and ours are related in that we are both studying the tendency of neural networks to rely on the simplest low-dimensional projection of inputs that fit the training data. Their theory leverages the literature on the infinite-width limit of 1-layer networks, whereas we directly study the bias-variance tradeoff in the finite-width setting using tools from information theory. Our understanding of the bias-variance tradeoff naturally suggests our method, which interpolates among orthogonal features and shows improved sample efficiency in practice.

Would the authors be able to provide a detailed analysis of the complexity for the proposed method (e.g. FLOPs and/or MACs)? Understanding complexity is crucial as it essentially represents the computational budget. How does the method's time complexity stand in comparison to leading fine-tuning approaches, such as BiT by Kolesnikov et al. (2020), which adapt the entire model body or FiT (Shysheya et al. 2022), which adapt a subset of the body parameters?

Our training involves at most two linear layers on top of cached feature vectors and is therefore generally much more computationally efficient than methods that adapt the entire model body or a large subset. The optimization problem is linear, so we were able to perform 30k runs within 24 hours using four standard CPUs and no GPUs.

We are happy to answer any further questions you may have. If we have addressed your concerns, we kindly ask that you may consider raising your score.