Mind the Gap Between Prototypes and Images in Cross-domain Finetuning

We post our responses regarding your concerns in the following:

Weakness 1: CoPA has two ... one linear classifier does not consider difference of two classifiers in gradients.

Answer: According to your concern, you also believe that a single transformation cannot consider the difference of gradients. That is what we wanted to highlight. The shared transformation is incompetent in CFC adaptation tasks.

In fact, the improvements of CoPA cannot be simply attributed to the complexity of models and loss.

In this paper,

• we first noticed that prototypes play a similar role to the text prompts in CLIP.
• Then, we empirically demonstrated that there exists a gap between the image and prototype embeddings, which is similar to the gap found in CLIP, and slightly enlarging the gap helps improve the generalization performance.
• According to further analyses, we found that the gap may be narrowed since applying a single transformation may damage the discriminative information in gradients.

This naturally motivated us to consider using two different transformations and substituting the text prompts with prototypes, as done in CLIP, to treat prototypes and images independently and perform representation alignment for better generalization performance. These points are highly appreciated by other reviewers.

Per your concern about complexity, as we have discussed in Section 5.2 (Line 330-338), applying decoupled transformations does increase the complexity of the hypothesis space, but it reduces the approximation error and is more flexible in representation learning (preserving the discriminative information). Meanwhile, we conducted several ablation studies in Appendix F3 & F5. As shown in Table 9, we perform URL with an MLP head (two 512$\times$512 layers). However, the performance drops evidently. This indicates that it is the flexible hypothesis space, instead of the model capacity, that facilitates the generalization performance. Besides, as shown in Table 10, although equipped with SCE loss (same as CoPA), URL fails to achieve similar performance to CoPA. This reveals that merely using SCE is insufficient for better generalization performance. Thus, the success of CoPA cannot be simply attributed to the complexity of the model and loss.

Weakness 2: There is no clear reason that ... such a uncertain phenomenon.

Answer: Per your concern, according to the paper, we noticed that prototypes play a similar role to the text prompts. Specifically, a prototype is the "common features" of a class of images, and a text prompt summarizes the "common sense" of a class of images. Thus, some semantic differences should exist between images and prototypes. According to empirical results, a gap does exists between prototypes and images. Such a gap describes the differences between prototypes and instances.

Meanwhile, as demonstrated in CLIP, for good generalization performance, representations of images and texts should be aligned. Thus, in Section 5.2, we argued that the essence of the gap enlargement is a kind of alignment between prototype and image instance representations, in which the optimal distributions of prototype and instance representations are explored. For a pre-trained model, where representations are well aligned, better generalization performance can be obtained.

Weakness 3: Also CoPA have a temperature parameter ... seems to be unfair...

Answer: Per your concern, the role of the temperature coefficient is tuning how concentrated the features are in the representation space. In fact, in practice, the NCC objective also adopts temperature coefficient, though it is not mentioned in the paper.

In the source code of URL (see Line 149 in ./models/losses.py file of our submitted supplementary file), a temperature coefficient 10 is multiplied to the logits. When reproducing URL, we adopted all original settings for fairness.

Question 1: This paper discuss the gap ... Thus, I feel strange that instances and prototypes are in the separate regions as shown in Fig.1. The number of samples per classes will influence the gap $\Delta$. Could you justify this issue?

Answer: Per your concern about the gap, we discuss it from two different angles. Firstly, if representations are not normalized, the prototype is within the convex of instances and influenced by the number of samples. Specifically, when the number of samples in each class is equal, the gap is 0.

However, according to [32], the calculation of the gap is based on normalized embeddings/representations (we have highlighted this in line 174-175). Here, we take the embedding gap as an example.

Consider a set of unnormalized enbeddings $\{f\_{\phi^*}(\boldsymbol{x}\_i)\}\_{i=1}^{|\mathcal{D}\_{\mathcal{T}}|}$, the prototype for class $c$ is $\boldsymbol{c}\_{c}=\frac{1}{|\mathcal{C}\_c|}\sum\_{f\_{\phi^*}(\boldsymbol{x})\in\mathcal{C}\_c}f\_{\phi^*}(\boldsymbol{x})$, where $\mathcal{C}\_c=\\{f\_{\phi^*}(\boldsymbol{x}\_i)|y\_i=c\\}$. Then, the gap vector is calculated as $\vec{\Delta}=\frac{1}{|\mathcal{D}\_{\mathcal{T}}|}\sum\_{i=1}^{\mathcal{D}\_{\mathcal{T}}}\frac{f\_{\phi^*}(\boldsymbol{x}\_i)}{||f\_{\phi^*}(\boldsymbol{x}\_i)||\_2}-\frac{1}{N\_C}\sum\_{j=1}^{N\_C}\frac{\boldsymbol{c}\_j}{||\boldsymbol{c}\_j||\_2}$ (see P7 of [32]). It is easy to see that the embeddings will be normalized when calculating the gap. In this case, the prototype may not be in the convex of images, and the relationship between the number of samples of each class and the gap is undetermined.

To further solve your concern, we checked the gap between prototype and instance embeddings under the 'vary-way 5-shot' setting, where the number of samples of each class is equal. It is easy to see that the gap still exists.

Dear Reviewer eGag,

Thanks again for your time and efforts in reviewing our work.

This is a gentle reminder that the Discussion period will end on 13 Aug 11:59 pm AoE. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors

Dear Reviewer eGag,

Thanks again for your time and efforts in reviewing our work. Do you still have any other concerns or questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors

Dear Reviewer eGag,

Thanks again for your time in reviewing our work.

This is a gentle reminder that the Discussion period will end on 13 Aug 11:59 pm AoE. There is only one day remaining. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors

Dear Reviewer eGag,

Thanks for your time and efforts in reviewing our work.

We have posted responses to your new concerns. Would you mind checking the responses and confirming whether the updated responses solve your concerns? We are glad to answer any concerns and questions you may have before the deadline.

As a gentle reminder, the discussion period will end in about 12 hours. We hope to resolve your concerns about our work.

Moreover, if your concerns have been resolved, would you mind reconsidering your rating of our work based on the updated understanding?

Best wishes,

Authors

Thanks for your effort and time in reviewing our work. We appreciate a lot for your positive evaluation towards our work!

For your concern about the novelty of our paper, we think that our contributions can be summarized as following:

• We first noticed that prototypes play a similar role to the text prompts adopted in the multi-modal domain, and conjectured that the phenomenon that there is a gap between the prototype and image instance embeddings/representations may exist.

• Then, we empirically validated the gap between prototypes and image instances. According to our empirical results, we found that

  • applying a single transformation head as done in previous work may narrow the gap,
  • the generalization performance can be improved by slightly enlarging the gap between the prototype and image instance embeddings.

• To further explore the gap, we conducted a series of theoretical analyses. The results reveal that

  • applying the same transformation potentially removes the discriminative information in gradients during the adaptation phase;
  • the upper bound of the representation gap limits the change of the gap when applying a single transformation head.

• Based on the observations and inspired by the previous work, we followed CLIP and proposed to use two decoupled transformations and substitute the text prompts in the SCE loss with prototypes to preserve the gap between the prototypes and image instances. Thus, we proposed an efficient and effective CFC adaptation method, CoPA.

• According to our experimental results, CoPA is able to

  • improve the generalization performance without hurting the efficiency，
  • preserve the gap between prototype and image instance representations,
  • explore the optimal distributions (representation space) to align the representations for better performance.

A potential novelty of our proposed CoPA method is that CoPA provides a new perspective of representation learning. Specifically, we can consider manually designing some kinds of data that contain different levels of information from the original data. Then, by performing representation alignment based on the flexible hypothesis space, we can learn better representations and achieve better performance without extra data. In this way, we can take advantage of the existing data more sufficiently and effectively.

Dear Reviewer PX84,

Thanks again for your time and efforts in reviewing our work. We appreciate your valuable and insightful comments on our work. We believe your expertise will help improve the quality of our work.

This is a gentle reminder that the Discussion period will end on 13 Aug 11:59 pm AoE. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors

Thanks for your effort and time in reviewing our work. We appreciate a lot for your insightful and valuable comments towards our work. Now, we post our responses regarding your concerns in the following:

Weakness 1: For the empirical analysis in Section 3.2, there is an interesting phenomenon that “appropriately enlarging the gap between the prototypes and image ...”. It may be better to provide some discussions of the reasons for these these observed results (Figure 3(a)) to explain the influence of “enlarging the gap” on the “generalization ability”.

Answer: Thanks for your interesting question. As we have mentioned in the paper, we found that the prototypes play the similar role to the text prompts used in multi-modal domains. Specifically, a prototype is the "common features" shared within a class of images, and the text prompt summarizes the "common sense" of a class of images. From this perspective, there should exist some kind of semantic differences between prototypes/text prompts and image instances. As shown in Fig. 1 and other visualization results in Appendix B, there do exist a gap between the prototype and image instance embeddings. Such a gap can be seen as the difference between two sets of data.

According to previous works, such as CLIP, to achieve good performance, the representations of images and texts should be aligned. Thus, as we argued in Section 5.2, we think the essence of the gap enlargement is a kind of alignment between prototype and image instance representations, in which the optimal distributions of prototype and instance representations are explored. Thus, for a pretrained model, where representations are well aligned, better generalization performance can be obtained.

Weakness 2: The competing methods used in experiments appear to be somewhat outdated. The main baseline URL is published in 2021. It may be better to compare with some recently proposed adaptation-based methods.

Answer: Thanks for your suggestion. In fact, we have compared CoPA with some recent representative adaptation-based baselines in our paper. Specifically, we considered TSA (CVPR 2022) and TA$^2$-Net (ICCV 2023), which achieved better empirical performance by leveraging extra adaptation learning modules, in Table 1 & 2. According to the results, CoPA outperforms these baselines evidently. We will continue to review other works and consider adding them as references in our updated version.

Weakness 3: Some equation numbers are missing. Some references are not correctly formatted, the conference or journal names as well as page numbers are missing.

Answer: Thanks for your suggestions. We will add equation numbers to all equations and modify the reference format in our updated versions.

Question 1: In Theorem 3.1, optimizing the first term in the equation between the 206th and 207th rows requires to maximize the similarities between instances and their corresponding protypes. So, would enlarging the protype-image gap negatively affect this optimization process?

Answer: Thanks for your insightful question. We would like to discuss this question via our existing empirical results. First of all, according to Fig. 3(a), it is easy to observe that the gap cannot be enlarged infinitely and the enlargement is limited in a small range. The validation loss tends to increase when the gap is larger than a threshold. This is reasonable since significantly enlarging the gap, as you mentioned, reduces the similarities between instances and their prototypes. Meanwhile, narrowing the gap (increasing the similarity) also results in the increase of validation loss, as excessively maximizing the similarity may lead to generalization problems (e.g., overfitting).

Moreover, according to Fig. 5(c), the minimal validation loss is achieved at the learned representation gap. In this case, both enlarging and narrowing the gap increase the validation loss. Since SCE can align different sets of representations and preserve the gap between two sets of data, we argued that the essence of the enlargement of the gap between image and prototype representations is exploring the optimal function space for representation alignment (see Section 5.2). Thus, we think such a slight enlargement will not negatively affect the optimization process.

This question is interesting. We will add this discussion in our updated version.

Question 2: According to the equations between the 218th and 219th rows, I can understand the differences of the first term in these two equations are removed, when the shared transformation is adopted. But why this may lead to the drop of “the discriminative information in gradients”? Could the authors explain this more clearly?

Answer: Thanks for your question. When two transformations are applied, the gradients for image instance and prototype transformations are respectively calculated with the equations mentioned in the paper. As you can see, the two gradients are different.

However, when applying a single transformation ($\Theta_{\rm I}=\Theta_{\rm P}=\Theta$), the gradient of $\Theta$ is $\nabla\_{\Theta}\mathcal{L}(\Theta)=-\frac{2}{|\mathcal{D}\_{\mathcal{T}}|}\Theta^{\top}f\_{\phi^*}(\boldsymbol{X})^{\top}YY^{\top}f\_{\phi^*}(\boldsymbol{X}) + \frac{2\alpha}{|\mathcal{D}\_{\mathcal{T}}|}\Theta^{\top}f\_{\phi^*}(\boldsymbol{X})^{\top}f\_{\phi^*}(\boldsymbol{X})$, where

• the first term is the combination of the first terms of $\nabla_{\Theta_{\rm P}}\mathcal{L}(\Theta_{\rm P}, \Theta_{\rm I})$ and $\nabla_{\Theta_{\rm I}}\mathcal{L}(\Theta_{\rm P}, \Theta_{\rm I})$ with $\Theta_{\rm P}=\Theta_{\rm I}=\Theta$,
• the second term is the second term of $\nabla_{\Theta_{\rm I}}\mathcal{L}(\Theta_{\rm P}, \Theta_{\rm I})$ with $\Theta_{\rm I}=\Theta$.

It is easy to find that the discriminative information of gradients is damaged since the gradient information regarding prototypes and image instances is mixed.

Dear Reviewer Ms7f,

Thanks again for your time and efforts in reviewing our work. We appreciate your valuable and insightful comments on our work. We believe your expertise will help improve the quality of our work.

This is a gentle reminder that the Discussion period will end on 13 Aug 11:59 pm AoE. Would you mind checking our responses to your concerns and questions and confirming whether you have any other questions? We are glad to answer any further questions you may have before the deadline.

Best regards,

Authors