Learning with Analogical Reasoning for Robust Few-Shot Learning

Thank you for the insightful feedbacks. We are working on more experiments on more dataset. In the following discussions, we first address part of your concerns.

W2: Limit of applicability in domains lacking such resources or where these resources are incomplete

A2: Our work depends on an existing knowledge base in which the semantically analogical relationships of concepts can be utilized to refine noisy prototypes. So, yes, our work has limitation as pointed out. However, we would like to emphasize that our work can be easily generalized to domains where no human curated knowledge base is ready, by leveraging an AI-powered one. For example, we can use CLIP, BGE, text2vector or other vector models that are trained supervised or unsupervised to construct a pseudo knowledge base. These models learn from abundant corpus and, to a certain extent, capture how we human perceive the world as well.

We use CLIP to build such an AI-powered knowledge base for noisy FSL on CIFAR-FS dataset. Experiments below show that AI-powered knowledge bases can be seamlessly integrated with our method. we will clarify this compatibility in the revision.

FSL with symmetric label swap noise and paired label swap noise on CIFAR-FS dataset.

Model \ Nois Proportion	0%	20%	40%	60%	40%(paired)
Oracle	79.99	78.12	75.32	69.20	75.32
ProtoNet	79.99	74.40	61.87	44.43	55.91
TarNFS (ours)	81.17	76.59	69.72	57.68	66.02

W3: The method may require considerable computational resources, which are not addressed in terms of efficiency or scalability.

A3: We investigate the efficiency of our method as in the table below when we do experiments on CIFAR-FS dataset. As can be seen, our method is as efficient as ProtoNet, especially during inference. The transformer-based architecture for AR ( the 2nd row) and LSTM architecture for task-level representations (the 3rd row) introduce negligible amount of parameters, and slightly increase the forward/backward time from a task averaged perspective.

Though LSTM in TCL increase the training time by about 20% per epoch, it introduces only 1.5% more time latency during inference when compared to ProtoNet. This is because TCL is enabled during training only and disabled once the training is finished.

FSL training/testing time and efficiency on CIFAR-FS (base model is ResNet12).

Model \ Time	#parameters	forward time/task	backward time/task	training time/epoch	testing time (100 tasks)
ProtoNet	12.42M	0.05276s	0.00971s	16.031s	4.09s
TarNFS (wo/TCL)	14.06M	0.05362s	0.01111s	16.1057s	4.16s
TarNFS (ours)	15.90M	0.07902s	0.02096s	19.447s	4.15s

W4: Adapting this method for domains with different types of semantic structures could present challenges.

A4: As discussed in A1, our method is compatible with both human-curated knowledge base (like WordNet) and AI-powered ones. For WordNet, we get relevant concepts for analogy-based reasoning via LeaCock-Chodorow similarity. For other domains with different types of semantic structures, we could find another metric (eg cosine similarity, euclidean distance, etc) for the similar purpose. The adaptation process is straightforward.

We use WordNet as the knowledge base in our work because it is widely utilized in the literature. It does not mean that our method can only use WordNet for noisy FSL. Other human-curated or AI-powered ones can be applied. We will make a revision to clarify the compatibility with other knowledge bases of our method.

We are doing experiments on MD. We will report the results together with comparisons to DEAT as soon as possible.

Regards

We have made a revision to address your concerns. Specifically,

1. To test the model’s real-world applicability on additional datasets where the knowledge base may be incomplete or missing, we conduct experiments on CIFAR-FS. We use ResNet12 as the learner so as to additionally test the model's applicability to new architecture. CIFAR-FS has categories that are not in WordNet which hinders our fast adaption to the new scenario. To this end, we use CLIP to build connections between categories which are then utilized as concept connections in a large but unknown knowledge base in our method. Experiments in Tab7 show that our method can easily generalize to this new challenging situation, boosting performance especially in the heavily noisy settings. Experiments on MiniImageNet with Conv4-64 (Tab1) and ResNet12 (Tab9) also demonstrate that our method generalizes well to different architectures.

2. We conduct experiments to investigate the efficiency of our method. Tab5 and Tab10 show that our method cause insignificant computational cost during training especially when the learner is large. During inference, since TCL is discarded, our method experiences no efficiency issue and runs as fast as ProtoNet in different settings.

3. We thank the reviewer for recommend DETA for comparison. DETA relies on in-class and out-of-class region similarities to estimate the weights of these regions. Weights of the regions are further aggregated into the weights of images and utilized in local compactness loss and global dispersion loss for model optimization. As can be seen, DETA can be significantly impacted by noisy samples, especially in heavily noisy settings. This is because, in our opinion, the weights of regions deteriorate seriously as the noise ratio increases, which hinders model optimization and leads to performance degradation.

We are open for discussion if you have any new concern.

As the discussion phase is coming to the end, could you please take a look at our response and see if your concerns are properly addressed. If not, we are still open for discussion.

We sincerely thank the reviewer for the helpful feedbacks. We address your comments bellow:

W1: Details of knowledge base construction and selection for analogical reasoning.

A1: In our method, we use a knowledge base from which we select relevant known categories for analogical reasoning to defend prototypes from noisy samples. The knowledge base we intend to use (as stated in Sec1 line080) is WordNet. For the construction of WordNet, please refer to https://wordnet.princeton.edu/. Note that other knowledge bases are also applicable in our method. For instance, when lacking of such a resource or the resource is incomplete (as the case pinpointed by Reviewer gFtz ), we can use CLIP or other embedding models like BGE, text2vector to construct a pseudo knowledge base in which the connections among categories can be estimated via cosine similarity metric.

To be specific, we construct a pseudo knowledge base for noisy FSL on CIFAR-FS dataset using CLIP and demonstrates that our method works well in the situation. See more details in A3.

W2: Discussion on training time and efficiency.

A2: TCL utilizes the pair of positive tasks and those negative tasks in the task queue to perform task-level contrastive learning for enhancing generalization. Since representations of negative tasks are computed in previous training iterations, the additional computation cost is from the construction of representations of positive tasks, and is negligible. We compares the forward time and backward time of ProtoNet, our TarNFS without TCL and TarNFS in the following table

FSL training/testing time and efficiency on CIFAR-FS (base model is ResNet12).

Model \ Time	#parameters	forward time/task	backward time/task	training time/epoch	testing time (100 tasks)
ProtoNet	12.42M	0.05276s	0.00971s	16.031s	4.09s
TarNFS (wo/TCL)	14.06M	0.05362s	0.01111s	16.1057s	4.16s
TarNFS (ours)	15.90M	0.07902s	0.02096s	19.447s	4.15s

As can be seen, LSTM in TCL slightly increases the number of model parameters and the forward/backward time.

Note that TCL is utilized in training stage only and is dropped during inference, introducing no efficiency concern in inference. Thus, our TarNFS is just a little bit slower than ProtoNet but greatly boosts the performance under different noisy settings.

W3: Scalability with larger or different types of knowledge bases.

A3: Our method is compatible with other knowledge bases, both the human curated (like YOGO) and the AI-powered (like CLIP, BGE, text2vector etc). WordNet is a good choice for miniImagenet and tieredImagenet due to that it contains all the categories in the two datasets. For experiments on datasets that such a knowledge base does not exist or is incomplete, we can use CLIP or other embedding models like BGE, text2vector to construct a pseudo knowledge base as describe in A1.

For example, we use CLIP to build a pseudo knowledge base that is utilized by our method on CIFAR-FS dataset. Results are as follows

FSL with symmetric label swap noise and paired label swap noise on CIFAR-FS dataset.

Model \ Nois Proportion	0%	20%	40%	60%	40%(paired)
Oracle	79.99	78.12	75.32	69.20	75.32
ProtoNet	79.99	74.40	61.87	44.43	55.91
TarNFS (ours)	81.17	76.59	69.72	57.68	66.02

Experiments demonstrate that our method scales to work in combination with the knowledge base well.

We hope that we have addressed all your concerns in our response. A revision with changes that clarifies those discussed above is coming.

We sincerely thank the reviewer for recognizing the contribution of our work. We address your concerns below.

W1: " additional results could be demonstrated using other datasets".

A1: As suggested, we conduct experiments on the CIFAR-FS dataset to demonstrate the effectiveness of our TarNFS. On CIFAR-FS, we note that not all categories are included in the knowledge base (i.e. WordNet) as pointed out by Reviewer gFtz. To perform analogy reasoning, we use CLIP to estimate the similarities or connections between novel categories and known categories. Specifically, for each category, CLIP takes "A picture of [category]" as input and represents the [category] as a vector. We then use cosine similarity to find associated known categories for each novel category, which builds a pseudo knowledge base that can be alternatively utilized in TarNFS.

Results are as follows:

FSL with symmetric label swap noise and paired label swap noise on CIFAR-FS dataset.

Model \ Nois Proportion	0%	20%	40%	60%	40%(paired)
Oracle	79.99	78.12	75.32	69.20	75.32
ProtoNet	79.99	74.40	61.87	44.43	55.91
TarNFS (ours)	81.17	76.59	69.72	57.68	66.02

Ablation study on CIFAR-FS in noisy setting below demonstrate the effectiveness of each component in our TarNFS.

PN	AR	TCL	0%	20%	40%	60%
$\surd$			79.99	74.40	61.87	44.43
$\surd$	$\surd$		80.37	74.54	65.81	51.12
$\surd$	$\surd$	$\surd$	81.17	76.59	69.72	57.68

We donot have enough time to complete experiments on Meta-Dataset yet. MD is of size 200GB. Moreover, the noisy setting on MD is different from those previously studied in the literature[1-2].

Since our experiments on miniImagenet, tieredImagenet and cifar-fs datasets showed the effectiveness of our method, we hypothesize that our method works on MD too.

[1] Liang, Kevin J., et al. "Few-shot learning with noisy labels." CVPR. 2022.

[2] Li, Junnan, et al. "Learning to learn from noisy labeled data." CVPR. 2019.

W2: Code for reproducibility

A2: To facilitate reproducibility, we will make the code public available on github. For double-blind review, our code can be temporarily found here.

Q: TarNFS outperforms the oracle under a 20% noise setting.

A: Oracle stands for ProtoNet of our implementation that only uses clean support images for FSL ( as described in Sec5.3). As listed in Table1 in Sec5, our TarNFS not only outperforms Oracle under clean FSL setting, but is also slightly better than Oracle under 20% noisy FSL setting on miniImagenet. This is due to that our TarNFS can learn more robust prototypes be leveraging analogy reasoning. But we do not insist that our method can consistently surpass Oracle in noisy settings. We note that the performance of FSL is indeed seriously impacted by noisy samples in the support set. In noisy settings, our TarNFS is generally inferior to Oracle as expected.

As requested by the reviewer, we here provide a task with 20% noise ratio that our method performs better than Oracle. Click to view.

Note: we do not indicate any insights by the illustration above, nor do we encourage readers or reviewers to hallucinate so. This is because FSL must be tested on a bunch of tasks. The performance on one task means nothing.

Q: Impact of using the TCL module alone.

A: In our TarNFS, we use prototypes before and after AR to construct a pair of positive task representations for TCL. It is difficult to analyze the impact TCL without considering AR. To isolates the impact of TCL as requested, we sample two FSL tasks from the same set of categories as a pair of positive tasks and disable the AR component. Results on CIFAR-FS w/wo AR are as follows:

FSL with symmetric label swap noise on CIFAR-FS dataset.

Model \ Nois Proportion	0%	20%	40%	60%
Oracle	79.99	78.12	75.32	69.20
ProtoNet	79.99	74.40	61.87	44.43
TarNFS(wo/AR)	81.09	74.90	63.21	44.30
TarNFS (ours)	81.17	76.59	69.72	57.68

As can be seen, TCL alone can boost performance when compared to ProtoNet, especially under clean setting and 40% noisy setting. And, our TarNFS that combines AR and TCL obtains much more better results in all settings.

Hope we have addressed your concerns. More experiments on MD is in processing. We will give a revision as soon as possible.