Distribution Calibration For Few-Shot Learning by Bayesian Relation Inference

First of all, we would like to thank the reviewer for their encouragement to our work. In this work, we propose a calibration distribution method based on Bayesian relation reference method for few-shot classification task. The highlight of the proposed model is that it achieves better performance compared to the SOTA few-shot learning algorithms, while visual analysis proves that the proposed model has a certain degree of interpretability, which is useful especially for medical scenarios such as dermatological classification.

Thanks very much for pointing out the weaknesses of the article. It helps us to better improve the deficiencies in our paper. We answer the questions raised by the reviewer below:

• Q1: What is the definition of the relation graph? What are the nodes and edges?

  A1: Thank you for raising this issue. In each relation graph in this paper( Summary graph $\boldsymbol{M}$ & Gaussian relation graph $\boldsymbol{A}$ et.al.), nodes are either base or target class embeddings, and the edges represent the relation intensity between target class and each base class. Specifically, target class node embedding $N\_{t} \\in \mathbb{R}^{\rm{N}\_{f}}$ is the feature generated by the backbone network( In our paper, we introduce pre-trained Resnet 18 as the backbone network) from a single target image. For the $i$-th base class node embedding $N_{b_{i}} \in \mathbb{R}^{\rm{N}_{f}}$, we first generate the features of all the images in the $i$-th base class through the backbone network, after which we take the average of the features as the base class node embedding.

• Q2: What is the range of the mean in Eq. (1)?

  A2: Thank you for pointing out this problem which we overlooked. We rewrite Eq. (1) as follow to make the expression clearer:

  ​ $$ \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\boldsymbol{N\_{b\_{i}}} = mean(\\sum\_{X\_{Y=y\_{i}}}\boldsymbol{f}\_{\operatorname{Res18}} (X\_{Y=y\_{i}})) \tag{1}

First of all, we would like to thank the reviewer for their encouragement to our work. In this work, we propose a calibration distribution method based on Bayesian relation reference method for few-shot classification task. The highlight of the proposed model is that it achieves better performance compared to the SOTA few-shot learning algorithms, while visual analysis proves that the proposed model has a certain degree of interpretability, which is useful especially for medical scenarios such as dermatological classification.

Thanks very much for pointing out the weaknesses of the article. It helps us to better improve the deficiencies in our paper. We summarize the weaknesses and the improvements based on these weaknesses as follows:

• W1: The model can be experimented on datasets other than the dermatology dataset.

  I1: Thank you for your constructive comment. The proposed Bayesian distribution calibration model is a generic model which can achieve satisfying performance on various datasets. We chose the Dermnet dataset for our experiments due to the fact that the model has a certain degree of interpretability, which is more important for medical scenarios. To further validate the performance of the model, we will experiment the proposed model on other datasets. Due to time as well as arithmetic issues, we choose the MiniImagenet dataset for the additional experiments, and the experiment result will be published in the supplementary material, thank you!

• W2: If required, we can generate the imbalanced datasets from the original dataset. More generalization and advantages are widely explored in the various fields, and if possible, the conventional algorithms for few-show learning can be added to baseline algorithms.

  I2: Thank you for your suggestions for experiment. Adding the conventional algorithms for few-show learning to baseline algorithms is a fascinating experiment that can provide some implications. We will compare the performance of the proposed Bayesian distributional calibration method with the conventional Resnet18 model on Dermnet dataset, and the experiment result will be published in the supplementary material soon, thank you!

• W3: The presentation is not kind for the reader, especially not familiar with few-shot learning and mathematical formulation of variation inference.

  I3: Thank you for pointing this out, which is an oversight in our writing. We made extensive revisions to the Method section (Section 3 Bayesian Relational Inference on page 3~7) of the manuscript to allow the reader to understand the model details more intuitively. Specifically, we made the following revisions to the Method section:

  • For some variables, we indicate their dimension. The description of the variable is modified to be similar to the following form:

    $\\boldsymbol{N_{t}} \\in \\mathbb{R}^{\\rm{N_{f}}}$ represents the target node embedding, $\\boldsymbol{N_{b}}\\in \\mathbb{R}^{\\rm{N_{b}} \\times \\rm{N_{f}}}$ represents the base node embeddings

  • We rewrite the assumptions related to the Bernoulli distribution as well as the computational part of the paper, and the physical meanings represented by variables are more precise. The revised presentation is similar to the follow form:

    $\\mathrm{L}\_{\\mathrm{mean}}(\\cdot)$ and $\mathrm{L}\_{\mathrm{std}}(\cdot)$ are implemented by neural networks for estimating the mean and standard deviation of the Gaussian distribution $\mathcal{N}\left(\mu_{i},\ \sigma_{\mathrm{i}}^{2}\right)$ which can calculate the approximation of the Bernoulli distribution $\mathcal{B}(n, \lambda_{\mathrm{i}})$ with $n\rightarrow \infty$ and $\lambda_{\mathrm{i}} \rightarrow 0$

  • We rewrite the ill-expressed formulas. For example:

    ​ \\begin {equation} \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\\widetilde{\\alpha}\_{\mathrm{i}}= \sqrt{(m\_{\mathrm{i}} \times (1.0 - m\_{\mathrm{i}}))} \times \varepsilon\_{\mathrm{i}}+m\_{\mathrm{i}} \tag{6} \end{equation}

  • For the theorems used in the paper we proof them in the supplementary material A.3.

Thanks very much for the suggestions and questions raised by the reviewer. We answer the questions raised by the reviewer below:

• Q1: Please clarify the notation $\mathcal{B}(n, \tilde{\lambda}_{\mathrm{i}})$ in the equation (16) since there is no definition.

  A1: Thanks for pointing out this issue. We did not clarify this Bernoulli distribution clearly in the paper previously. This is an assumption for summary graph mentioned in Section 3.1, where we assume that each edge of the summary graph is a sampling of a Bernoulli distribution $\mathcal{B}(n, \lambda)$ with $n\rightarrow \infty$ and $\lambda \rightarrow 0$. We modify the expression in Section 3.1 to explicitly state the assumptions associated with that Bernoulli distribution. Besides, we explicitly state the basis for conversion to these two KL terms above equation (16) in Section 3.3. The modifications to Section 3.1 and above for equation (16) in Section 3.3 are as follows, respectively:

  In section 3.1:

  Hence we assume that the summary graph of the coupling is a sampling of a Bernoulli distribution $\mathcal{B}(n, \lambda)$ with $n\rightarrow \infty$ and $\lambda \rightarrow 0$.

  In Section 3.3: Since each variable in $\mathbf{S}$ is affected by $\tilde{\mathbf{A}}$ in Eq.(6), we have $s\_{i} \mid \tilde{\alpha}\_{i} \sim \mathcal{N}\left(\tilde{\alpha}\_{i} * \mu\_{i}, \tilde{\alpha}\_{i} * \sigma\_{i}^{2}\right)$. Besides, each element in Gaussian graph is conditioned on the Binomial variable for the same edge of the summary graph. Hence The $\mathrm{KL}$ term can be further written as:

  ​\begin{aligned} &\sum_{\mathrm{i}}^{\rm{N_{b}}}\Bigg \\{ \\mathrm{KL}\bigg(\mathcal{B}(n, \lambda_{\mathrm{i}}) \| \mathcal{B}\left(n, \lambda_{\mathrm{i}}^{(0)}\right)\bigg) \\\\ \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad +\mathbb{E}\_{\mathrm{\tilde{\alpha}}\_{\mathrm{i}}}\bigg[\mathrm{KL}\bigg( \mathcal{N}&\left(\tilde{\alpha}\_{\mathrm{i}} * \mu\_{i},\ \tilde{\alpha}\_{\mathrm{i}} * \sigma\_{\mathrm{i}}^{2}\right)) \| \mathcal{N}\Big(\tilde{\alpha}\_{\mathrm{i}} * \mu\_{i}^{(0)}, \tilde{\alpha}\_{\mathrm{i}} * \sigma\_{\mathrm{i}}^{(0)^{2}}\Big)\bigg)\bigg]\Bigg\\} \end{aligned}

• Q2: In equation (15), I cannot understand the $\sum_{i=1}^{M}$ since there is not explicit $i$ in arguments of $p$ and $q$. Please clarify this issue.

  A2: Thank you for raising this issue. $M$ is the number of Gaussian relation graphs generated in one batch. We add to our paper the note on the meaning of $M$. Equation (15) is now as follow:

  ​$\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \sum\_{i=1}^{M}\bigg(\mathrm{KL}\left(q\left(\tilde{\mathbf{A}}\_{i}, \mathbf{S}\_{i} \mid \mathbf{N}\_{\mathrm{t}}, \mathbf{N}\_{\mathrm{1}: \mathrm{n\_{b}}}\right) \| p\left(\tilde{\mathbf{A}}\_{i}, \mathbf{S}\_{i} \mid \mathbf{N}\_{\mathrm{t}}, \mathbf{N}\_{\mathrm{1}: \mathrm{n\_{b}}}\right)\right)\bigg)$

In few-shot learning algorithms, training a model using conventional algorithms can be difficult due to the large number of categories and the fact that the data for most of the categories is scarce. We divide the test set of the Dermnet dataset in a ratio of 8:2 into a new training and test set. Then we generate a conventional algorithm model by replacing the few-shot classification part of the Bayesian relational inference model( Multi-view Gaussian graph generation component and logistic regression classifier) with a linear classification head. We freeze the Bayesian relation inference module and fine-tune the classification head on the new training set and test it on the new test. We compare the accuracy of this algorithm with that of the few-shot learning algorithms on the 5-way 1-shot task. The experiment result is shown as follow:

Specifically, we adopt a three-layer artificial neural network as the linear classification head of the conventional algorithm, trained for 1500 epochs using the Adam optimizer with the learning rate of 0.0008. The result shows that the conventional algorithm's accuracy is similar to that of the early few-shot algorithms on the 5-way 1-shot task. It is worth noting that traing the conventional algorithm is time-consuming and overall performs less well than the few-shot algorithms.

The task of skin disease classification is famous and important. However, there are other datasets concerning ImageNet, Food-101 and so on. We further perform experiments on miniImageNet for few-shot classification tasks to validate the effectiveness of the proposed Bayesian relational inference model.

Due to the space constraints of the main text, we would like to devote more space to the proposed Bayesian distribution calibration method, so in the revised manuscript we only add the results of the two SOTA models on the Dermnet dataset. It is worth noting that the performance of our BDC model on the 5w5s task is further improved during the rebuttal process (from 68.58% to 70.03%). In addition, we have added the rest of the experiments, which are posted in the Supplementary Material due to space constraints, and if we have the opportunity, we will reorganize the paper structure in the camera ready version and organize some of the experiments (e.g., the results of the models on the miniImageNet dataset) in the main text manuscript.

Thank you very much for your suggestions on the experiment part of the paper, we have revised the paper and optimized the wording of the Section Related Work and Method. The experiment result of the coventional algorithm compared with the few-shot algorithms is shown in the manuscript. The experiment content of the paper is now more complete. Subsequently, we plan to add other additional experiments, which will be completed before the final review comments due to arithmetic issues. The experiments we expect to add are as follows:

• Test the robustness of the proposed Bayesian distribution calibration model by extracting features using backbone networks other than Resnet18, e.g., VGG16 and Resnet101.
• Train classification algorithms other than logistic regression classifiers for few-shot classification tasks, e.g., SVM and decision tree algorithms, to verify the performance of the fusion features obtained through the Bayesian relational inference model on different classification algorithms.

First of all, we would like to thank the reviewer for their encouragement to our work. In this work, we propose a calibration distribution method based on Bayesian relation reference method for few-shot classification task. The highlight of the proposed model is that it achieves better performance compared to the SOTA few-shot learning algorithms, while visual analysis proves that the proposed model has a certain degree of interpretability, which is useful especially for medical scenarios such as dermatological classification.

Thanks very much for pointing out the weaknesses of the article. It helps us to better improve the deficiencies in our paper. We summarize the weaknesses and the improvements based on these weaknesses as follows:

• W1: Some notations are confusing.

  I1: Thank you for your constructive comment. We revised the manuscript, changing notations that were ambiguous to the paper. Specifically, we replace the notation "$\rm{N}$", which denotes the dimension, with "$\rm{K}$", and harmonize the representation of σ throughout the text. as follows:

  $\rm{K_{b}}$ represents the number of base class nodes, $\boldsymbol{N_{t}} \in \mathbb{R}^{\rm{K_{f}}}$ represents the target node embedding,

  ${\\sigma}\_{\mathrm{i}}=\operatorname{\zeta}\left(\ \mathrm{L}\_{\mathrm{std}}\left(\boldsymbol{E}\_{\mathrm{e}\_{\mathrm{i}}}\right)\ \right)$

• W2: The term "Bernoulli distribution" used in the paper is actually the binomial distribution.

  I2: Thank you for pointing out our clerical error, we have corrected the part of the manuscript that was incorrectly written as "Bernoulli distribution" to "binomial distribution".

• W3: It needs to take some effort to understand the logic in some paragraphs.

  I3: Thank you for your suggestion for our paper. In previous revisions, we have endeavored to make the formulas in the manuscript less difficult to understand, but some of the mathematical transformations used (e.g., the transform from binomial distribution to Gaussian distribution) could not be recounted in detail due to space limitations in the article. We show these theorems in the supplementary material and marked in the manuscript (e.g., Please refer to the supplementary material A.3.2 for the calculation of this value in Section 3.3, page 6). In addition we have made some wording changes to improve the readability of the article.

• W4: A previous paper on distribution calibration [1] provides some visualizations of the distribution that can facilitate understanding (Figure 1, Figure 2, and Figure 6). It would be helpful to provide some visualizations.

  I4: Thank you very much for your suggestion, the article you mentioned is an inspiring paper and we will refer to it to enrich the content of our manuscript and make it more understandable. Due to time constraints, these changes will be finalized after the rebuttal and uploaded to the anonymous GitHub.

Thank you for your response! I have a clearer understanding, but I am still confused about a few issues.

1. I am still confused about how $m_i$ is derived. By comparing the theorem and Equation 5, it seems that $l = 2 \sigma_i^2 / (1 - 2 \mu) = 2 n_i \sigma_i^2$, which leads to $n_i = \frac{1}{1-2\mu}$. How is it obtained?
2. The writing still needs to be clearer, for example, the A.1.3 subsection in the appendix should explain what the hyper-parameters are.
3. Can you specifically describe how you optimize the model? Do you train the classification head and the Bayesian relational inference component together or sequentially? During the validation and test phase, I guess you fix the Bayesian relational inference component and simply optimize the logistic regression classifier, is my understanding correct? If you train the logistic regression classifier on the validation or test set, is there data leakage, and do the baseline methods also optimize some parts of the neural network on the validation and test set?
4. You claimed in the conclusion that "to the best of our knowledge, this is the first attempt to combine the Bayesian relational inference method with distribution calibration." One of the main novelties you state is that the proposed method can automatically calculate the relation score, while previous methods usually used manually designed distance measurements. Is there any previous method that also calculates the relation automatically? If there are some, could you please explain your uniqueness?