Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning

To Reviewer HqaU :

Weaknesses (1)

Response 1 : About computational overhead.

1. Thanks for the comments. The proposed method indeed introduces additional computational cost during the training phase. We will further clarify this weakness in the manuscript.

2. We present the computational costs of our method during the training and inference phases in Table 2 and 3 (Due to character limitations, please see "To Reviewer SF8W".), respectively. As shown, the backbone and the feature reconstruction network in our method are lightweight, which to some extent mitigates the computational overhead of the high and low-frequency branches. In addition, a portion of the computational overhead of our method comes from the FFT. In future work, we will explore the use of efficient, learnable image decomposition techniques, employing differentiable kernels to capture information from different frequency bands of the images.

3. During the inference, the proposed method does not introduce additional computational overhead, resulting in efficient inference and good generalization.

Response 2 : About the reconstruction prior.

We have analyzed the reconstruction prior in the manuscript (please see "Effectiveness of the proposed frequency prior" in 4.3. Ablation study.) and performed experiments (please see Table 3 in the manuscript). We copy the results from the manuscript here, as shown in Table 12. It can be seen that compared with the baseline, the proposed reconstruction prior can achieve better performance. This demonstrates the contribution of the proposed reconstruction prior.

Table 12 : Verifying the alignment and reconstruction prior under 5-way 1-shot (5-way 5-shot ) setting.

Weaknesses (2)

Response :

Thanks for the suggestion. We will supplement the references for CropDisease, CUB, and ISIC in the manuscript.

Weaknesses (3)

Response :

Thanks for the comments. We will further clarify this weakness in the manuscript. To alleviate this problem, we will explore the use of learnable image decomposition methods as an alternative to FFT. Essentially, FFT uses fixed kernels for signal decomposition, making it difficult to achieve data-adaptive decomposition. We are considering designing differentiable kernels to separately extract high-frequency and low-frequency information from images. Additionally, we consider constructing a model that generates kernel parameters conditioned on the original image for data-adaptive image decomposition. We will validate it in future work.

Questions (1)

Response :

We think that it is valuable to study cross-domain few-shot learning (CD-FSL) in the era of foundation models. The reasons are as follows.

1. Adapting foundation models to CD-FSL has practical value. The core elements of CD-FSL are feature initialization and rapid adaptation. Early research has concentrated on using meta-learning methods to learn a good feature initialization model in the source domain, and then quickly adapting the model to the target task with a small number of labeled samples.The foundation models essentially accomplished the first step of FSL. In future research, effectively adapting large models to cross-domain few-shot tasks is valuable.

2. Exploring the adaptation of foundation models to extreme CD-FSL holds significant research value. Previous work [1] explored applying pre-trained foundation models to extreme cross-domain few-shot tasks. For comparison, we implemented our method under the same settings, and the results are shown in Table 13. As can be seen, even with the assistance of pre-trained foundation models, these methods still perform poorly in extreme cross-domain scenarios (e.g., Chest, ISIC, etc.). Therefore, exploring the adaptability of foundational models to extreme CD-FSL tasks is valuable.

Table 13 : Performance using the foundation model as feature initialization.

[1] Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference, CVPR 2022.

Questions (2)

Response :

Following the suggestion, we have fine-tuned our method on target domain. Following the common fine-tuning strategies, for each few-shot task in the target domain, we fine-tune the model using the support set, and then test it on the query set. Due to time constraints, we have conducted experiments on the EuroSAT and ISIC target datasets as examples, as shown in Table 14. It can be seen that, through fine-tuning, the performance of the proposed method can be further improved. For example, on the EuroSAT dataset, fine-tuning the proposed method improved performance by 1.55%.

Table 14 : Performance when fine-tuning our method under 5-way 5-shot setting.

To Reviewer mWuv :

Weaknesses (1)

Response :

We present the computational costs during the training and inference phases in Table 2 and 3 (Due to character limitations, please see "To Reviewer SF8W".), respectively. We can draw the following observations.

1. During the training, the computational overhead of our method mainly comes from the backbone and image decomposition. As shown in Table 2 (please see "To Reviewer SF8W"), the backbone and the feature reconstruction network are lightweight, which to some extent mitigates the computational overhead.

2. During inference, our method does not introduce additional computational overhead, resulting in efficient inference and good generalization.

Weaknesses (2) & Questions (5) & Questions (6)

Response 1: Using Resnet-101 and ViT as backbones.

We validate our method with Resnet-101 and ViT as backbones, as shown in Table 6 (Please see Table 6 in the global rebuttal). It can be seen that our method effectively scales to different backbones and demonstrates certain performance advantages.

Response 2: Verifying cosine loss.

We validate cosine similarity as an alternative to MSE, as shown in Table 7 (Please see Table 7 in the global rebuttal). It can be observed that MSE and cosine similarity yield nearly equivalent performance.

Response 3: About KL divergence.

In our alignment prior, we use the predictions from the main branch as an anchor, and expect the high/low-frequency predictions to align with this anchor. To achieve this, we use KL divergence as the metric. The reasons are as follows. First, since the anchor is dynamically changing during training, meaning the entropy of the anchor is not constant, cross-entropy is not suitable. Second, our alignment loss is asymmetric, so JS divergence is also not applicable. In addition, Wasserstein-1 distance could serve as an alternative to KL divergence. However, it requires computing the optimal transport between different prediction distributions, which involves quadratic programming and results in higher computational complexity.

Weaknesses (3)

Response :

Thanks for the suggestions. In this work, we resort to the image decomposition prior which have been proved shared across different images despite their domains, i.e., each image can be decomposed into complementary low-frequency content details and high-frequency robust structural characteristics. More importantly, we specially establish a feature reconstruction prior and a prediction consistency prior to separately encourage the consistency. This allows for collectively guiding the network’s meta-learning process with the aim of learning cross-domain generalizable embeddings. We will supplement more theoretical analysis into the manuscript.

Questions (1)

Response :

We validate the hyper-parameters in FFT(e.g., radius ratio), which determines the boundary between high and low frequencies. The results are shown in Table 8 (Please see Table 8 in the global rebuttal). It can be observed that this hyper-parameter does not significantly affect the overall performance. We set it to 0.5 for all cases.

Questions (2)

Response 1 : About training epoch.

We validate the impact of training epochs on performance. The results are shown in Table 9 (Please see Table 9 in the global rebuttal). It can be observed that as the number of epochs increases, performance initially rises and then declines. However, since we cannot link to the test data during the training phase, we can only judge based on the loss curve during training (please see Fig. 2 in Response.pdf). We observed that the loss tends to converge around 50 epochs. Therefore, we set it to 50 for all cases.

Response 2 : About loss curve and landscape.

We provide the loss optimization curve and loss landscape in Fig. 2. As can be seen, the baseline tends to over-fit at the relatively early epochs. In contrast, with the assistance of prior regularization, our method can mitigate over-fitting to some extent. For the loss landscape, we followed the implementation in [1]. Firstly, randomly perturb the model trained in the source domain in 2000 directions; Secondly, perform inference on the target domain for each perturbed model, and record the loss value; Finally, we visualize the loss landscape based on the recorded loss values and directions. It can be seen that, compared to the baseline, the trained model of the proposed method is more robust to unknown perturbations. For example, the loss values in our method converge within a certain range. In contrast, the baseline method's loss values tend to diverge. This demonstrates the good generalization ability of our method.

[1] Visualizing the loss landscape of neural nets. NIPS, 2018.

Questions (3)

Response :

We designed a variant experiment that utilizes the raw support set to predict high-frequency and low-frequency query. The results are shown in Table 10 (Please see Table 10 in the global rebuttal). It can be seen that this variant also achieves significant improvements over the baseline. However, the performance of this variant is slightly lower than our original approach. This may be because independently using high-frequency or low-frequency support images to predict high-frequency or low-frequency query images can more explicitly utilize the diverse frequency information.

Questions (4)

Response :

We supplement some experiments to verify alternative mechanisms, and the results are shown in Table 11 (Please see the global rebuttal). We compared three update mechanisms. Case 1: we keep parameters shared among the three branches. Case 2: non-shared parameters between the three branches and joint training in an end-to-end manner. Case 3 (Ours): EMA strategy to update. As can be seen, our method achieves the best performance.

Questions (7)

Response :

Thanks for the suggestions. We will supplement those results into the main text.

To Reviewer SF8W:

Weaknesses 1

Response :

We compared the FFT decomposition results of natural images and EuroSAT images (please see Fig. 1 in Response.pdf). We observed that FFT decomposition of natural images can obtain clear low-frequency and high-frequency information. However, the high-frequency part of the image from EuroSAT is almost all noise. There may be two reasons for this. One is that during the imaging process of remote sensing scene images, due to the high spatial resolution of remote sensing scene images, the camera often compresses high-frequency information to improve imaging efficiency. Second, remote sensing scene images focus on containing low-frequency texture content and lack clear high-frequency structural information.

We will further clarify this weakness in the manuscript.

Weaknesses 2

Response :

Following the suggestion, we have supplemented experiments for the proposed method in the in-domain setting. All in-domain experiments followed standard settings, the results as shown in Table 1. It can be seen that the proposed method achieved good results with both Resnet-10 and ViT-S backbones.

We will supplement these experimental results into the manuscript.

Table 1 : Performance on within-domain few-shot learning tasks under 5-way 5-shot setting.

[1] Prototypical networks for few-shot learning， NIPS 2017.

[2] Meta-learning with differentiable convex optimization，CVPR 2019.

[3] Matching Feature Sets for Few-Shot Image Classification, CVPR 2022.

[4] DiffKendall: A Novel Approach for Few-Shot Learning with Differentiable Kendall's Rank Correlation, NIPS 2023.

[5] MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning, AAAI 2024.

[6] Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference, CVPR 2022.

Weaknesses 3

Response :

Following the suggestion, we present the computational costs of the proposed method during the training and inference phases in Tables 2 and 3, respectively. We can draw the following conclusions.

1. The backbone network and the feature reconstruction network in our method are lightweight, which to some extent mitigates the computational overhead.

2. In addition, a portion of the computational overhead of our method comes from the FFT. In future work, we will explore the use of efficient, learnable image decomposition techniques, employing differentiable convolutional kernels to capture information from different frequency bands of the images.

3. During the inference phase, the proposed method does not introduce additional computational overhead, resulting in efficient inference and good generalization.

We will supplement these analyses and discussion into the manuscript.

Table 2 : About parameters (M), FLOPs (G), and Iteration time (S) during training phase. Take the 5-way 1-shot 15-query task as an example to calculate the floating point and training iteration time. N represents the number of pixels in each image.

Table 3 : Performance and efficiency during target domain inference phase under 5-way 1-shot (5-way 5-shot ) setting.

Weaknesses 4

Response :

Following the suggestion, we compared the performance and efficiency under different decomposition methods. The results are shown in Tables 4 and 5. It can be seen that, compared to the baseline methods, the proposed method achieves better results across various decomposition methods. Additionally, using wavelet-based decomposition results in higher computational efficiency. When we prioritize efficiency, wavelet decomposition is the optimal choice. However, compared to FFT, the performance of wavelet decomposition is slightly lower. Therefore, overall, FFT remains a better choice.

We will supplement these analyses and discussion into the manuscript.

Table 4 : Performance on different image decomposition method under 5-way 1-shot (5-way 5-shot ) setting.

Table 5 : Decomposition efficiency (sec).