Independently-prepared Query-efficient Model Selection

Thank you for reviewing our paper!

We are glad to know that you like both the new model selection paradigm and the idea of standardization. We also apologize as you find some description hard to understand: We are here to address any questions from you and will use them to improve the clarity and quality of our paper during revisions.

1. I don't understand "any subset of the baseline feature set B can be directly associated with a binary vector $\\{0,1\\}^N$."

Please allow us to explain this with a toy example. Assuming now the baseline feature set contains a total of $N=4$ features, $b_0, b_1, b_2, b_3$, then there will be a total of $2^4=16$ different subsets of B, and we can associate each subset with a distinct, 4-dimensional binary vector (i.e. a vector in $\\{0,1\\}^4$). Specifically, (0, 0, 0, 0) will denote the empty subset, (0, 0, 1, 0) will denote $\\{b_2\\}$ and (1, 0, 1, 1) will denote $\\{b_0, b_2, b_3\\}$, i.e. each 1 indicating the presence of a feature and each 0 indicating an absence.

2. I think there may be a typo related to $\delta$-equivalence.

Good catch! This is indeed a typo and we will have it fixed: The optimization maximizes cosine similarity and therefore it is trying to find a subset of the baseline features that is $\delta$-equivalent to F for smaller $\delta$.

3. Why is $v$ constrainsted in $[0, 1]^n$?

Because $v$ is a continuous relaxation of a binary vector from $\\{0, 1\\}^n$. As we clarified in part 1 of our response for you, each subset of the baseline features can be associated with a binary vector from $\\{0, 1\\}^n$ (i.e. every dimension is either 0 or 1). The idea of continuous relaxation is to use $v$, a vector with each dimension constrained to be any value between 0 and 1, as a surrogate for binary vectors defined in Definition 2 so that the overall optimization problem will be continuous instead of discrete.

4. Why is the equivalence defined using affine transformations? For example, why don't we use $w^\top F(x)$ only?

That is a good question! Conceptually, how we should define equivalence of features depends on our knowledge/assumption regarding how these features will be used. Using affine transformations is because in most use cases of pre-trained features, there will be at least one linear layer and at least one linear layer will have a learnable bias term. For example, say now there will be one linear layer $g_{w_g, b_g}(x) = w_g^\top x + b_g$ attached when using features (there may be more layers after g), then we have $g_{w_g, b_g}(w^\top F(x) + b) = g_{w w_g, b_g + w_g^\top b}{F(x)}$, meaning that any functions representable by a linear layer (with bias) after $w^\top F(x) + b$ will also be representable by a linear layer (with bias) after $F(x)$. We will include explanations like this in our paper to make it easier to understand.

5. Are there any theoretical justification for the standardization? Although it's intuitive, the theoretical analysis is necessary to explain the effectiveness.

We consider our proposed method, standardized embedder, as an theoretically motivated approach, meaning that it is derived through theoretical tools and theoretical analysis but there is no rigorous theoretical guarantee of effectiveness. Respectfully, we argue ‘a rigorous theoretical guarantee of effectiveness’ is valuable but not necessary, given that no existing model selection work (as we discussed in Section 2 Related Work) offers a guarantee of such.

6. The comparison between Standardized Embedder and other existing model selection methods are missing

The main subject of this paper is NOT standardized embedder BUT the new paradigm, i.e. independently-prepared query-efficient model selection. As we emphasize in our paper, standardized embedder is a proof-of-concept solution to showcase that the new paradigm is possible.

The primary benefit of the new paradigm (compared to existing ones) is that it has fundamentally better preparation and query complexity & scalability with respect to increasing numbers of candidate models. These originate from the paradigm design and are elaborated in Section 3 of our paper.

Consequently, the purpose of our proof-of-concept solution and our experiments is to show that such a paradigm is realizable and feasible. Since existing model selection methods do not fit in the new paradigm, we argue that the existing experiments should be sufficient for this purpose.

Thank you for responding to our rebuttal!

---

(1) For 1&2, we just updated the manuscript. Note that for 1, we add the illustrative example to Appendix C due to space limits and refer to it in Section 4.2.

---

(2) For 3&4&8&11, thank you for letting us know that these are addressed!

---

(3) For 5&6&7&9, we wonder if the reviewer is currently considering only a narrow definition of impact that excludes our primary contributions. To be specific, we admit that the current evaluations only support the effectiveness of standardized embedder in the context of image classification and linear probing, however, we argue that it is more than sufficient as a proof-of-concept to support the feasibility of the new paradigm and more importantly, a new paradigm without a robust technique does NOT necessarily limit its impact.

Here is an example to illustrate the structure of scientific revolutions: It is well-known that the Wright brothers invented the first airplane in 1903, which is certainly a paradigm changer since it demonstrates the possibility of heavier-than-air, sustained, powered and controlled aircraft. However, this plane is not really a robust technique, given that it survived only 4 brief, low-altitude flights with a maximum distance of 852 ft (260 m) before it got damaged in landing and minutes later blew over by a heavy gust. It was not until 36 years later, in 1939, that we had the first jet aircraft, and not until another 13 years later, in 1952, that we had the first jet airliner.

Obviously, we are not claiming that our new model selection paradigm is as important as the first airplane or the future progress based on our paradigm will be as difficult as inventing the first jet aircraft. We just hope more diverse definitions of contributions can be considered. In the context of this submission, we argue that the impact of the submission is already sufficient since the new paradigm offers benefits that are critical to applications and the proof-of-concept solution supports the feasibility/realizability of the new paradigm.

---

(4) In addition, for 5, if possible, could you please provide an example reference of model selection with the degree of theoretical justifications that you find sufficient? We believe provable model selection can itself be a very promising research project and will be very interested in any relevant work.

---

(5) For 10: The short answer is simply that one can consider different feature layers of the same pre-trained model as different models. If you recall, we define each model as a set of features when introducing our method in Section 4. For example, if I have a pretrained model containing two layers $f_1: \mathcal{X} \to \mathbb{R}^{d_1}$ and $f_2:\mathbb{R}^{d_1} \to \mathbb{R}^{d_2}$, then I simply can have two different feature sets depending on which layer will be used as features: One is the composition of all layers $f_2\circ f_1: \mathcal{X} \to \mathbb{R}^{d_2}$, and another one simply contains a single layer $f_1: \mathcal{X} \to \mathbb{R}^{d_1}$. They can be treated as two separate models when using standardized embedder.

---

Thanks again! Please let us know if anything. Your prompt feedbacks will always be appreciated!

Dear reviewer,

We sincerely appreciate your active participation in the discussion!

We will cover the points following the order they are mentioned in your latest reply.

---

For point (5): ‘I'm interested in understanding how to select the most effective feature layer from a pre-trained model for optimal performance.’

We rephrase the question a little bit: Given a task and a pre-trained model, how can one select a layer of the pre-trained model that is the most useful for this task? Obviously, the answer depends on both the design of the model and the task. A common practice nowadays is to use the penultimate layer, because it is found to be working fine in many cases (which is why we follow this in our experiments). However, obviously this is not necessarily the optimal choice and in many cases so far, people have to select a layer based on some heuristics: For example, if it is known that the task depends heavily on spatial information (e.g. similar objects/features in different locations may mean different things), one should probably use features before pooling instead of the penultimate layer (where location information is removed by global pooling) for a ResNet model.

One can still select layers based on heuristics for model selection, but if one prefers automatic solutions, our method may provide a solution (by embedding multiple layers and selecting as if they are different models). This can be another potential benefit of Standardized Embedder (even though we do not emphasize it in our paper as it is relatively minor).

---

For point (4): My query is not about the theoretical underpinnings of model selection, but rather about the specific justifications for your proposed "Standardized Embedder" method.

Could you please advise on why you find the theoretical justification of Standardized Embedder insufficient? We are a bit confused on this front, because we believe we offer a similar/higher level of theoretical justifications as existing work. For example, Task2Vec/Model2Vec [1] provides a fairly similar level of theoretical justifications as ours in a sense that their method is also derived with theoretical motivations but with no theoretical guarantee of success. We believe Task2Vec/Model2Vec [1] is typically considered a good work in this field.

reference: [1] Achille, Alessandro, et al. "Task2vec: Task embedding for meta-learning." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

---

For point (3):

Thank you for the detailed response and your new comments! We will start addressing them for you now.

1.How vital is privacy preservation in your paradigm? Once a model is selected through your algorithm, full access for training is necessary. How do you address this in practical applications?

To clarify, the proposed paradigm do not remove the requirement of (eventually) accessing the selected model. As detailed in Section 3.2, the privacy benefits are twofold: (1) Model owners only share embeddings and their models are kept private during selection. (2) The users (who want to select models for their tasks) can keep their data to their own and they do not need to inform model owners about the selection. After a model is selected, the user contacts only the owner of the selected model to gain access (for example, to purchase the model).

Comparing to existing paradigms, an important value of ours is that model owners do not need to share their models to any others just to get considered in model selections: In the case of distributed model markets, this means that model owners only grant access to actual buyers (that have selected their models) and potential buyers do not need to purchase models they do not want just for model selection.

2.Converting a model into embeddings might entail loss of significant model information due to compression.

Indeed, this is exactly why we build a proof-of-concept solution to show that it is possible to keep sufficient information to select models that perform well. Obviously, we are not saying that every method following the new paradigm should be effective. What we argue is that there are methods that can be effective while enjoying the benefits of the new paradigm and therefore the new paradigm is promising.

Thank you so much for getting back to us.

Please allow us to provide additional information and clarifications for the concerns you just mentioned.

---

1. We agree with you that adding evaluations with fine-tuning can strengthen the experiments of this work. However, we want to add that fine-tuning is not necessarily the go-to option in practice and linear probing is still useful. For example, it is suggested that linear probing usually offers better out-of-distribution generalizations than fine-tuning, per the evaluation by Kumar et al.

reference: Kumar, Ananya, et al. "Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution." International Conference on Learning Representations. 2022.

---

2. To clarify, we do not describe the technique as a proof-of-concept solution for real-world model selection. We describe the technique as a proof-of-concept solution for our paradigm, or more specifically, it is a proof-of-concept for the following argument: One can learn model embeddings that are indicative of downstream performance while keeping the embedding process independent for different models.

---

3. To clarify, here is how we claim the advantage of our paradigm (and therefore our proof-of-concept solution) compared to existing ones: With a sufficiently large number of candidate models (say M candidate models), our paradigm (and our solution), is the only one so far that can scale. While transferability metrics require O(M) model operations per query and Model2Vec requires updating O(M) existing model embeddings when a new candidate joins, Standardized embedder requires a constant amount of (i.e. O(1)) model operations per query and update only 1 model embedding when a new candidate joins (which is the embedding corresponding to the new candidate). This is also why we believe the computation analysis we shared with Reviewer SGa4 should be sufficient to demonstrate our advantage in scalability.

---

Lastly, while obviously we as authors believe in our own work, we respect the reviewer's judgment and hope these new clarifications can be helpful towards a well-informed decision.

Thank you for your services!

Thank you so much for liking our paper! It is really a delight to have our contributions recognized!

Meanwhile, we will be doing our best in providing responses to your questions.

1. The existing work [1] seems to evaluate the pool of 1000s models. The scale of the number of models could be further increased.

[1] is already mentioned in our paper. Notably, they evaluate a pool of 65 models instead of 1000s models, despite the fact that they mention ‘100s and 1000s of source models’ when introducing their motivations. That being said, we still agree with you that further increasing the number of models can further strengthen our empirical results, at least to some extent. However, we also want to recap that the most important contribution of this submission is the new model selection paradigm rather than the proof-of-concept solution to support its feasibility. We believe the scale of current evaluations is sufficient in this context.

reference: [1] Bolya, Daniel, Rohit Mittapalli, and Judy Hoffman. Scalable diverse model selection for accessible transfer learning

2. A typo of strike-out text in Section 5.2.

Great catch! Thank you for reading our paper carefully!

3. What is the impact of using different models for baseline features?

This is a good question. We use Section 5.3 of our paper to discuss exactly the topic. In summary, from what we observe in Figure 5 and Figure 6, it is likely that models with attention, such as Swin Transformer (tiny) we evaluated, can be better choices of baseline features, because the reception fields of CNN may pose some limitations in characterizing some long-range correlations. However, it remains open regarding exactly what kind of models will serve best as baseline features (i.e. we know there are likely many good choices, including the ones evaluated, but we do not know yet which should be the best).

This is potentially one future direction for us.

4. What are other potential applications of this new paradigm of model selection?

The essence of the new paradigm is to enable independent embedding of different models, which makes the entire selection scheme updatable and decentralizable. Building on these, there may be potential applications like: (1) vectorized database for models; (2) web search engines that search model embeddings; (3) decentralized model markets (e.g. on blockchain). Of course this is unlikely to be an exhaustive list.

Thank you again for recognizing our contributions! Please let us know if the response helps. Any follow-up questions are welcomed!

5. The time consumption is not shown in the experimental results, please show the time consumption of each step and compare it with baselines, especially the time for model selection.

Since we did not ensure the consistency of hardwares (e.g. type and occupancy of GPUs) when running experiments, the actual time consumption is not available. However, we provide here the amount of computation required for each stage in our experiments, which can already showcase the scalability of the new paradigm. This information is also available in Section 5.1.

For preparation, we train either 4k steps or 10k steps per candidate model, which is equivalent to about 10 and 25 epochs on the ImageNet validation set (which contains 50k samples). Notably, during preparation, both the candidate model and the baseline features are kept frozen: Only two linear heads and an embedding vector will be updated. These computations are considered quite affordable, especially because each model only needs to be prepared once and different model owners can potentially handle the preparation of their own models. Notably, for the existing method Model2Vec (i.e. query-efficient model selection), each candidate model will also need to be evaluated on multiple predefined downstream tasks before learning all model embeddings jointly, which further limits their usability.

For queries, we use essentially the same training recipes as the final evaluation of downstream accuracy (i.e. linear probing with 60 epochs on the downstream dataset). Approximately and informally, if the amount of computation required to tune and evaluate one model for a given downstream task is $C$, then the primary computations required is $2C$ for standardized embedder and query-efficient model selection, while it will be $O(MC)$ (with some constant) with transferability metrics, where $M$ is the number of candidate models.

Thanks again for your review. Please let us know your comments at your earliest convenience. Best regards.

Thank you for your services. It is good to know that you find our perspective very interesting. Please allow us to provide clarifications regarding your comments.

1. The author says that he has proposed a new paradigm, but he has only made a little modification in the computation and mapping of embedded vectors, which does not match the innovation of the "new paradigm", and it is more suitable to use the "new method" for the discussion.

The core of this work is a new paradigm called independently-prepared query-efficient model selection, which indeed only differs from the existing query-efficient paradigm by requiring independent embedding of models. While this change is small, we respectfully argue that it is significant enough to be consider a new paradigm for the following reasons:

(1) This small change brings us many benefits: updatability, decentralizability, flexibility, and certain preservation of candidate privacy, in addition to certain preservation of query privacy inherited from the query-efficient model selection paradigm (as we presented in Section 3.2).

(2) None of the existing solutions remain applicable after this small change: Our solution, standardized embedder, is derived through a very different framework from the existing methods.

We do not know if you are familiar with self-supervised pre-training or not, but it may be a good example to explain our opinions. There is only a small change from the setting of semi-supervised training (i.e. assuming access to both labeled and unlabeled data) to self-supervised learning (i.e. assuming access to only unlabeled data): One simply removes the access to the labeled part. However, the idea of training on labeled data first and using the trained model to provide pseudo-label for unlabeled data is applicable only to semi-supervised settings but not self-supervised settings. We believe self-supervised learning is typically considered a different paradigm from semi-supervised learning.

2. The theoretical proof is not sufficient to strongly support the author's point of view, while for the author's so-called new paradigm I suggest to describe it in relation to a specific application scenario.

Sorry we are not really sure what your concern is here. Could you please explain a bit more regarding which point of ours is not sufficiently supported? Thank you!

3. Since it is a new paradigm, it should be suitable for all types of tasks and not just limited to deep models for object classification, or it should not be limited to the CV domain, but cover other such as NLP text classification, tabular classification, time-series classification and so on; Can the new paradigm be applied to other areas like NLP, time series analysis, etc.?

Notably, standardized embedder is just a proof-of-concept solution to support the feasibility of the new paradigm. While conceptually the new paradigm is generalizable to other domains like language, time series, tabular data and to other tasks like segmentation and detection, we choose to focus on demonstrating its possibility in image classifications, which is arguably one of the most popular settings in previous work studying model selection.

4. With the authors' experimental setup, I see that the claimed 100 pre-trained models are different hyper-parameter settings for several commonly used models, such as the number of network layers and so on. This is a different selection of models than what I understand, and I think the authors' experiments are more appropriately classified as either neural architecture search or hyperparameter optimization. Simply by modifying the hyperparameters should not be referred to as a different model.

We respectfully disagree that our evaluated candidate models are obtained ‘simply by modifying the hyperparameters’: While there are indeed some models with similar training recipes and similar architectures (i.e. they are obviously not 100 models with 100 distinct architectures and 100 distinct training recipes), but overall the candidate models we consider offer sufficient diversity regarding architectures (various ConvNets and Transformers) and training recipes (e.g. DINO, Masked Autoencoder, CLIP, Sharpness-aware minimizer). In addition, these models are pre-trained models from torchvision or timm, meaning that they are potentially trained by different teams and many people are actually using them in practice. We argue that while this set is not a perfect representative of every possible use case of model selection, it should be sufficient to support the evaluation of a proof-of-concept solution.