A Hierarchical Bayesian Model for Few-Shot Meta Learning

1. Similarity to most Bayesian meta-learning approaches in the context of few-shot learning.

The novelty of our work compared to other Bayesian approaches is four folds:

• 1. We capture uncertainty of entire model, including the full feature extractor, by treating every weight as a random variable and placing priors on them. Most existing Bayesian meta-learners only capture uncertainty of a subset of weights (EG: VERSA, MetaQDA, DKT) and rely on deterministic feature extractors.
• 2. Our Hierarchical Bayesian framework distinctly captures uncertainty at both the task ($\theta$) and dataset ($\phi$) levels. Some existing Bayesian attempts are not hierarchical and do not separately model these sources of uncertainty. This significance of hierarchical distinction is explored intuitively in Sec 4.
• 3. We achieve hierarchical Bayesian modeling of the full neural architecture (1+2 above) with low computational overhead thanks to the proposed closed-form episodic loss quadratic approximation. Our Meta-NIW is more efficient than both (non-Bayesian) MAML, and prior hierarchical Bayesian attempts such as PMAML. PMAML has only been demonstrated on small models and datasets, while we achieve full Bayesian uncertainty modeling on large VITs and challenging benchmarks such as ShapeNet2D and tieredImageNet.
• 4. We provide rigorous theoretical analysis to support the generalisation capability of the provided algorithm (Sec 5). This guarantee is also a novel result to our knowledge.

1. The claim of ”A novel hierarchical Bayesian model for FSL” in the abstract.

Thanks for the feedback. We agree that several other papers took hierarchical Bayesian perspectives on FSL. Our contribution lies in a specific parametrization and inference algorithm that allows fast and effective learning of such HBMs in large neural networks. We have adjusted our wording accordingly in the revised paper.

2. Describing some of the existing methods as improper solutions of the Bayesian few-shot learning problem.

Thanks for the feedback. We felt there were some limitations in existing attempts, such as being Bayesian but non-hierarchical, or only modeling a subset of the weights in a hierarchical Bayesian manner. However, we will carefully revise and tone down our discussion to avoid overclaiming. And of course, we have cited the mentioned papers that we did not include initially. Please check our revised paper (e.g., end of Sec. 6 Related Work).

3. The claim of "Bayesian treatment of only a small fraction of the network limits the benefits from uncertainty modeling". About CNP/ANP baselines to support claims on improved uncertainty modeling.

With regard to improved uncertainty modeling, we mention the CNP/ANP for their non-Bayesian modeling. In our evaluation on uncertainty calibration (Table 5), we did not compare ours to CNP/ANP but mainly to MetaQDA, a Bayesian model that is state of the art among those that only provide a Bayesian treatment of the output layer (VERSA, Gordon ICLR’19 and DKT Patacchiola’20, etc). The comparison here supports the claim that our Bayesian treatment of the entire network is beneficial.

4. Comparison to Bayesian Neural Process models. Relative merits of a complete Bayesian treatment.

Regarding the question on the relative merits of a complete Bayesian treatment but with restricted Gaussian forms versus a shallow Bayesian but GMM-like rich variational forms – We do not know which is absolutely better than the other. Further theoretical and empirical study needs to be carried out in this regard. But at this point, we have some experimental evidence, the comparison with MetaQDA shallow Bayesian approach that only places prior distribution on the model head parts while freezing the feature extractor (Table 2,3,5). As shown, our complete Bayesian treatment outperformed it, in both test generalisation performance and uncertainty calibration. This is one supporting evidence of why a complete Bayesian treatment could be more promising than a shallow Bayesian treatment.

For the experimental demonstration, we have now done additional experiments on the Sine-Line dataset to report the predictive marginal test likelihood score, which is directly comparable to the shallow Bayesian embedding models (e.g., [6]'s Bayesian NP model). Please see our table in response to Q7 below (or the second table in our responses to all reviewers).

5. Most of the experiments are classification tasks. Why ECE metric?

Our main target problem is few-shot meta learning (FSL). And those classification experiments (miniImageNet, tieredImageNet) are recognised as the mainstream benchmarks for FSL. We follow prior Bayesian FSL methods (eg: MetaQDA - Zhang ICCV’21; DKT - Patacchiola NeurIPS’20; Amortised Meta - Ravi ICLR’19) to assess uncertainty quantification via ECE and R-ECE on these standard benchmarks.

6. Calibration measures in Table 6.

Please refer to (Tran et al. 2020) and (Cui et al. 2020) . Together with the test likelihood score, we believe that the ECE score is the mainstream measure for uncertainty calibration (Hospedales et al., 2022; Wang et al., 2020a). We go beyond the evaluations of existing learners that mainly focused on uncertainty quantification in classification (ECE, Tab 5), and also report uncertainty quantification in regression (Tab 6).

7. Provide results in terms of predictive log marginal likelihood.

As requested by the reviewer, we have done experiments with the SineLine dataset to report the predictive log-marginal-likelihood scores (LMLHD). We try to match the experimental settings from [6] so that the results are comparable. More specifically, we test on 256 tasks, each of which consists of 64 samples with a varying number of context samples (1,2,4,6,8,12, and 16). The number of posterior samples used to compute/approximate the predictive marginal likelihood is 1024. The results are as follows (the higher the better uncertainty quantification and generalisation). As shown, our NIW-Meta has higher scores, especially for larger context size, implying that capturing uncertainty in full model parameters is important for generalisation capability.

8. What is the theoretical motivation for distributional estimate $q(\phi)$?

If $\phi$ were directly linked to the observed data $D_i$’s, then it is true that $\phi$ can tend to be determined deterministically, as the number of data $N$ becomes large. However, $\phi$ is linked to latent variables $\theta_i$’s, so the belief on $\phi$ also needs to capture/accumulate the uncertainty in $\theta_i$’s, which amounts to marginalising out the $\theta_i$ variables. So, it may not not be appropriate to treat the posterior for $\phi$ as a delta function (0 uncertainty), and it is better to follow the Bayesian inference principle, i.e., let the posterior be computed from the observed evidence.

9. Why do we need a distributional estimate $q(\phi)$ if at meta-test time we only use the mode of $q(\phi)$?

Using the mode of $q(\phi)$ is only for practical convenience and simplicity. Although we used the mode, the distributional form $q(\phi)$ would take into account uncertainty in its optimisation, and thus lead to a different solution from the deterministic one.

10. For meta testing, why the authors do not use the same inference approach as they use during meta training?

The method that the reviewer said also results in the same meta test solution, and this can also be verified by inspecting the similarity between Eq.(13) and Eq.(7 or 8). However, the mentioned approach requires that $L_0$ be fixed at the trained value, which we do not know for sure in the pure optimisation perspective. In Sec. 3.2, we aimed to derive the meta-test optimisation problem from the Bayesian perspective from the outset, which offers us a reasonable justification for why $L_0$ can be fixed.

11. Is the two-stage optimization approach (described in the paragraph before Eq.(7)) guaranteed to find the same solution as the original joint optimization problem?

Yes, the two-stage approach guarantees to find the same solution as the original joint problem. As we mentioned in the paper, this is essentially the same as $\min_{x,y} f(x,y) = \min_x f(x,y^*(x))$ where $y^*(x) = \arg\min_y f(x,y)$ for each given x – by the very definition of the joint minimisation problem.

12. Some more architectural details about how the combination of the authors proposed approach with set-based backbones like CNP works, cf. end of Sec. 3.

We basically follow the architectures from (Gao et al. 2022) https://arxiv.org/pdf/2203.04905.pdf For instance, for the ShapeNet1D case, the networks to which our NIW-Meta was applied are:

• CNP:

  • Rsupp = enc2( enc1(Xsupp), Ysupp ) where enc1 = 4-layer conv net, enc2 = 2-layer MLP
  • Zsupp = max-pool(Rsupp)
  • Yqry_prediction = dec( enc1(Xqry), Zsupp ) where dec = 3-layer MLP

• ANP:

  • We replace “max-pool” in CNP by “multi-head-attn( enc1(Xsupp), Rsupp, enc1(Xqry) )”

• C+R (Conv-Net + Ridge regression head):

  • Yqry_prediction = W* @ \phi(Xqry) where \phi() is 4-layer conv + 2-layer MLP, and
  • W* = argmin_W || W @ \phi(Xsupp) - Ysupp ||^2 + \lambda*||W||^2 (a closed-form solution)

For further details and other datasets, please refer to (Gao et al. 2022).

13. Not all results come with error bounds. The boldfaced results. Table 3/4 captions do not say that classification accuracy is reported.

• Error bars: Now we have added the SEMs for the results in Table 3 to show the statistical significance. Please see the revised paper for the full results and also the table below for only DINO/s CIFAR. Even though for some tasks there are some overlaps of the error intervals between our proposed approach and the competing methods, overall our approach is superior to the existing methods with statistical significance to some extent.

• Boldfaced results: We marked the ones with the largest average values.

• For Table 3/4: In our revised paper, we have mentioned that they are classification accuracy. Thank you for pointing this out.

1. Effectiveness of the multiple approximations, the ELBO and the quadratic approximation, in the proposed framework.

Both ELBO and the quadratic approximation are essential parts of our approach. ELBO is one of the few approximate methods to make Bayesian inference feasible, and is also used by other Bayesian methods such as VERSA (Gordon ICLR’19). And our quadratic approximation is what distinguishes it from other (hierarchical) Bayesian methods such as Probabilistic MAML (PMAML).

2. Empirical results for some tasks only show average values without confidence intervals.

We now have added the SEMs for the results in Table 3 to show the statistical significance. Please see the revised paper for the full results and also the table in our official comments for all reviewers for only DINO/s CIFAR. Even though for some tasks there are some overlaps of the error intervals between our proposed approach and the competing methods, overall our approach is superior to the existing methods with statistical significance to some extent.

3. Choice of the hyperparameters introduced by the proposed method.

Most hyperparameters are discussed and reported in Sec. D (Impl. details and experimental settings) of Appendix. For the test-time number of VI steps, we use 3 (mini/tiered-ImageNet), 2 (SineLine), and 20 (ShapeNets) iterations. The test-time number of posterior model samples is: 5 (mini/tiered-ImageNet and SineLine) and 2 (ShapeNets) samples.

1. More background knowledge for the NIW model and its justification.

The main reason why NIW model is adopted is that it allows a closed-form solution to the approximate episodic optimisation problem due to conjugacy with Gaussian. This is what enables us to achieve fast task-wise adaptation in (episodic) few-shot meta learning. This improved efficiency is part of what allows us to fully apply Bayesian learning to state of the art feature extractors to achieve excellent results, where many existing Bayesian attempts are restricted to relatively toy problems (PMAML, etc), or only a one Bayesian layer within a NNet (VERSA, MetaQDA, etc).

Regarding the background knowledge, we recommend and refer to K. Murphy's MLPP book (https://probml.github.io/pml-book/book1.html), especially the probability and Bayesian statistics sections (eg, Ch.2, Ch.4.6, and Ch.A.8.3). It is overwhelming for us to make the paper fully self-contained by including all required background materials in it. Instead, we have cited these background materials in the revised paper.

2. Advantage over hierarchical recurrent VAEs?

At first glance models following the Hierarchical Bayesian structures may look similar to each other. But the main difference is specific modeling and approximate inference strategies employed, in particular tailored for the specific FSL problem in this paper. We need fast task-wise adaptation, ie, fast solution for the inner variable variational optimisation, and to this end we came up with novel quadratic likelihood approximation under the NIW conjugate prior-posterior modeling.

Another novelty is the theoretical analysis, which to the best of our knowledge has not been done for other HBMs.

3. Interpretation in the sense of deep learning in general.

One can find interesting interpretations from our discussions in Sec.3.1, where we interpret various non-Bayesian methods including MAML, ProtoNet, and Reptile as special cases of ours. That is, these non-Bayesian methods can be instantiated by dropping either stochasticity (uncertainty modeling) or prior-related terms in our Bayesian model.

4. Generalisation ability of the model to tasks other than FSL?

Our approach can, in principle, be applicable to any other problems that involve multiple different data distributions (e.g., client/local data distributions in Federated or distributed learning). But we focus in this paper on the FSL problem mainly, leaving the application to other problems as future work.

1. What aspect of the proposed method improves the prediction performance? The Bayesian treatment?

Yes. We attribute our good performance to the Bayesian nature of our method. More specifically due to the three factors:

• Uncertainty capturing. Our ability to capture uncertainty is beneficial compared to non-Bayesian methods, but is shared with other Bayesian methods;

• Full Bayesian treatment. We treat all parameters in the whole network as random variables in Bayesian inference (IE: capturing uncertainty everywhere instead of only at certain layers). This improves performance compared to those Bayesian methods that are only Bayesian about a subset of layers. EG: MetaQDA (Zhang, ICCV’21); VERSA (Gordon, ICLR’19); DKT (Patacchiola, NeurIPS’20); etc. Importantly, the minority of other methods that attempt a Bayesian treatment of every parameter incur a substantial computational overhead (eg: PMAML, Grant ICLR’18), which limits their applicability to powerful modern neural networks required to achieve good performance. NIW-Meta's efficient Bayesian treatment enables us to benefit from both full uncertainty capturing together with state of the art feature extractors.

• Hierarchical Bayesian uncertainty capturing. Our hierarchical Bayesian model captures distinct uncertainty effects within tasks/episodes, and across tasks/episodes. This distinction is illustrated in the toy experiment of Table 1 and Figure 2, 3. Several existing Bayesian methods (eg: VERSA, etc) are not hierarchical in this sense, and so do not benefit from this.

2. Missing experimental comparison with other Bayesian meta-learning methods.

We did already compare with two Bayesian meta-learning methods: MetaQDA and PMAML. MetaQDA takes a limited Bayesian treatment (only one Bayesian layer in the NN), prioritizing the ability to scale to state of the art datasets and neural feature extractors and achieve state of the art results. MetaQDA already compares to, and beats, a range of prior Bayesian methods in their paper (VERSA - ICLR’19, DKT - NeurIPS20, etc), so we indirectly compare against even more Bayesian methods. Thus, we focus on comparing to MetaQDA as one of the strongest Bayesian SotA, which we outperform reliably (Tab 2, 3, 5). PMAML is a more complete Bayesian treatment, but does not scale to the large neural architectures required to achieve credible performance in vision tasks. Therefore we restrict our comparison to PMAML to few-shot regression problems (Sec 7.2).

The FSL paper identified by the reviewer, (Zhang et al. 2023), is an extension of the amortized Bayesian inference (Ravi and Beatson, ICLR’19), which aims to accelerate the slow computation time of the original method. Comparing with these other Bayesian methods, our accuracy is significantly higher, and calibration is similar or better. Please see the table below.

On miniImageNet (5-way, 1-shot) with Conv-4:

3. The calibration performance is validated using only a single data set.

Actually, we evaluated calibration on two datasets/tasks: Both miniImageNet (Tab 5) and Sine-Line (Tab 6). We have now also done new experiments to evaluate calibration in terms of the predictive marginal likelihood at test time. Please see our responses to EN3o’s Q.3(c).

4. What kind of information about the proposed method can we get from the final forms of the theorems, Eq.(14) and Eq.(15)?

The theorems say that our hierarchical Bayesian modeling and our approximate posterior family choice (ie, NIW), are sufficiently good enough for provable test performance. That is, our ultimate optimisation objective Eq.(6), once optimised properly, can lead to a model that generalises well to unseen test episodes with the provided certificates.

5. Comparing the actual training times in Fig. 5.

Figure 5 is actually already a fair comparison. Figure 5(b) already shows the full actual training times for competing methods. (Due to the far inferior performance of MAML, we are not able to compare the methods at similar performance.) However, we realise that there are some unclear points which may cause misunderstanding. For instance, in [Fig.5 (b) Left], we have two NIW-Meta models with different numbers of SGLD iterations (#SGLD=2 and #SGLD=5). The plots already include the time for both the number of burn-in steps, which is 2 burn-in steps in this case, and SGLD iterations (so, more precisely, they run 2+2 and 5+2 SGLD iterations, respectively, and compared to MAML with 1~5 inner iterations). It is our mistake not to mention this clearly enough, and we have clarified this in our revised paper.

1. Possible limitations of the approach for future work.

• Limitations: 1) NIW-Meta introduces some extra hyperparameters (eg, the number of SGLD iterations, the number of burn-in steps). These are currently estimated empirically, but a more rigorous study on how to select them automatically needs to be addressed. 2) Although it is empirically verified that our quadratic episodic loss optimisation is effective, more theoretical analysis on the quality of this approximation as well as its impact on the final results, needs to be done.

• Future works: We have quite an extensive evaluation on popular few-shot classification and regression benchmarks. However, we would like to evaluate our approach on new emerging applications of FSL such as efficient learning of implicit neural representations such as NERF, eg [A].

[A] Tancik et al, CVPR21, Learned initializations for optimizing coordinate-based neural representations.

2. Layout of the toy experiment section. Adding a proof sketch for Theorem.

• Toy experiment section: We can move the section after the theoretical analysis section. We will think about which way is the best for presentation.

• Proof sketch for theorems: We only put the final theoretical results in the main paper due to the lack of space. But we agree with the reviewer that it would be helpful to provide some overview or sketch of the proofs. We will figure out how we can include them concisely should the paper be accepted.

3. Choice of the NIW distribution. Limitations of using NIW.

The reason for employing NIW is mainly because it enables closed-form solutions to local episodic optimisation, which is essential for fast task adaptation in episodic FSL.

A Potential limitation of NIW might be that NIW is a single-modal distribution, thus of limited flexibility compared to the unrestricted, say nonparametric distribution families. For example, if mixing classification (object recognition) and regression (pose estimation) episodes which might prefer position invariant and position equivariant feature extraction respectively, then a uni-modal prior might be overly restrictive and multi-modal prior might be preferred. Still, it’s unclear how to perform efficient task adaptation with a multi-modal prior.

Impact of the NIW assumption compared to other FSL methods: As we mentioned in Sec.3.1 (Interpretation), those MAML, ProtoNet and Reptile are all shown to be special cases of our framework, even when we followed our NIW prior-posterior family. So there is no substantive limitation compared to existing FSL methods because of our NIW family choice.