Meta-Learning Universal Priors Using Non-Injective Change of Variables

Thank you for the questions. The issues raised are addressed one-by-one next.

Response to weakness 1

This contribution differs remarkably from probabilistic meta-learning in three key aspects.

• Deterministic algorithm. Our MetaNCoV is a deterministic meta-learning approach where task-level optimization (10b) is formulated as a maximum a posteriori (MAP) problem. This MAP estimator can be efficiently approximated by gradient descent. In comparison, probabilistic meta-learning methods aim to approximate the intractable posterior $p(\boldsymbol{\phi}_t | \mathbf{y}_t^\mathrm{trn}; \mathbf{X}_t^\mathrm{trn}, \boldsymbol{\theta}) \propto p(\mathbf{y}_t^\mathrm{trn} | \boldsymbol{\phi}_t; \mathbf{X}_t^\mathrm{trn}) p(\boldsymbol{\phi}_t; \boldsymbol{\theta})$ using variational inference [1,3] or particle sampling [2].
• Prior Expressiveness. While probabilistic meta-learning relies on tractable prior and (surrogate) posterior pdfs of prefixed forms, such as Gaussian [1,3], our approach forgoes the tractability for enhanced prior expressiveness. This allows for a data-driven prior of universal forms beyond the Gaussian family.
• Theoretical Analysis. The theoretical analysis also distinguishes the current paper with probabilistic meta-learning methods [1-3], which are based on empirical designs.

Response to weakness 2

We will add these three additional baselines to Table 1 for a more comprehensive comparison.

Response to questions and limitations

Our method is deterministic rather than probabilistic, which cannot directly handle uncertainty quantification due to the intractability of the prior pdf, as mentioned in Remark 3.3.

The complexity analysis and performance trade-off have been already investigated in our submission, with results respectively reported in Appendices D.4 and D.3. It can be seen that our MetaNCoV has comparable time and space complexity relative to popular optimization-based meta-learning approaches.

[1] S. Ravi, and A. Beatson, "Amortised Bayesian meta-learning," ICLR 2019.

[2] J. Yoon, T. Kim, O. Dia, S. Kim, Y. Bengio, and S. Ahn, "Bayesian model-agnostic meta-learning," NeurIPS 2018.

[3] C. Finn, K. Xu, and S. Levine, "Probabilistic model-agnostic meta-learning," NeurIPS 2018.

Thank you for the clarifying answers, which have addressed some of my questions. I have some additional questions: Could you please elaborate more on the "prefixed forms" for the above-mentioned probabilistic baselines? Specifically, is the Gaussian assumption imposed both at the prior and posterior level? Moreover, what is the rationale for not considering the above-mentioned methods as important baselines for comparison?

Thank you for your follow-up questions, which are addressed in the following.

Regarding prefixed priors in probabilistic meta-learning

In [Section 3.3, 1], the surrogate (variational) posterior is prespecified as a diagonal Gaussian distribution, while the prior is fixed to be the Gaussian-Gamma form. For [2], particle sampling with SVGD is utilized to parameterize and optimize the prior, where RBF kernels (i.e., Gaussian kernels) are selected in Section 5 to interpolate the particles. As also noted in [Section 4.2, 3] and [Algorithm 1, 3], both the prior and surrogate posterior distributions are predefined as diagonal Gaussian forms.

Regarding the exclusion of probabilistic methods

The primary rationales for excluding these probabilistic methods is their high computational complexities compared to deterministic ones. In particular, [1] and [3] rely on variational inference, which requires sampling from the surrogate posterior $q(\boldsymbol{\phi}_t; \mathbf{v}_t)$ (with variational parameter $\mathbf{v}_t$) at each optimization step to calculate the expected NLL $\mathbb{E} _{q(\boldsymbol{\phi}_t; \mathbf{v}_t)} -\log p (\mathbf{y}_t^\mathrm{trn} | \boldsymbol{\phi}_t; \mathbf{X}_t^\mathrm{trn})$. In [2], a group of particles is maintained to parameterized the parameter distribution. As a result, these probabilistic methods [1-3] suffer from $M\times$ time- and space-complexity compared to deterministic ones, where $M$ is the number of samples.

However, it is worth noting that our MetaNCoV consistently outperforms these probabilistic methods by a significant margin. Please refer to the table below for a comparison on the 5-class miniImageNet dataset.

Thank you for providing the insightful suggestions. The concerns are addressed one-by-one as follows.

Regarding Weakness 1

Our Sylvester NCoV intentionally forgoes the injectivity constraint to enhance prior expressiveness, as illustrated in Theorem 3.1. Upon waiving the injectivity, the function $f$ is no longer bijective and thus can be non-invertible. Consequently, the Jacobian $J_f$ needs not to be a positive definite triangular matrix.

Regarding Weakness 2

The claim in line 256 aims to illustrate that both MetaNCoV and LEO optimize a latent variable instead of the primal $\boldsymbol{\phi}_t$. We agree with the reviewer that our latent variable is designed from a different perspective. We will rewrite this sentence to emphasize this difference and avoid potential confusion.

Regarding Weakness 3

Our method distinguishes from [1] in three key aspects:

• Prior to learn. The proposed MetaNCoV aims to learn the prior $p(\boldsymbol{\phi}_t)$ over model parameters $\boldsymbol{\phi}_t$, while [1] propose to learn the prior $p(g(\mathbf{x}_t^n))$ over the extracted features $g(\mathbf{x}_t^n)$. Additionally, [1] relies on preselected prior form (i.e., conjugate prior) to ensure tractability, whereas our goal is to meta-learn a data-driven prior of universal forms.
• Task-level optimization. This work belongs to the optimization-based meta-learning, which finetunes the entire nonlinear model during task-level optimization. In contrast, [1] is a metric-based approach that freezes the feature extractor and adapts merely the last linear layer of the model.
• Randomness. Our approach is deterministic (cf. maximum a posteriori (10b)), focusing on enhanced prior expressiveness without necessitating a tractable pdf. In comparison, [1]'s task-level optimization seeks the probabilistic posterior distribution, which concentrates on uncertainty quantification and pdf tractability.

Therefore, [1] is parallel to our work and not directly comparable. Thanks for pointing out this related work; these differences will be highlighted in Appendix E of the revised paper.

[1] X. Zhang, D. Meng, H. Gouk, and T. Hospedales, "Shallow Bayesian meta learning for real-world few-shot recognition," ICCV 2021.

I thank the authors for their efforts in addressing my concerns in terms of weaknesses 1 and 2. I still hold different ideas about the comparison with [1] but I will change my score to borderline accept.

Thanks for the valuable feedback provided. The questions raised are addressed one-by-one below.

Regarding weaknesses 1 and 2

We would like to kindly bring to the reviewer's attention that the model complexity with meta-parameter dimension $D$ has been already studied extensively in Appendix D.4; additionally, related works have been provided in Appendix E of the original submission. Moreover, we have referred to them in lines 327 and 193 of the main paper to guide the readers to relevant Appendices. As evident from Table 6 in Appendix D.4, parameters of our MetaNCoV is notably less than MC, and its time- and space-complexity is comparable to popular optimization-based approaches. Furthermore, while NFs are thoroughly studied, the NCoV models have gained little attention in machine learning and statistical learning due to their intractability; see lines 701-706 in Appendix E for related works and corresponding comparisons.

Regarding weakness 3

Our main claim in this work is that MetaNCoV empowers more expressive (rather than scalable) prior in high-dimensional spaces; cf. lines 12, 64, 176, and 188. This argument has been also empirically confirmed in our numerical tests in Section 4 and Appendix D. In terms of scalability, MetaNCoV is comparable to popular optimization-based approaches such as MAML and MC; see Table 6. However, it is still challenging to directly apply these algorithms (even vanilla MAML) to extremely large-scale models such as vision transformers. Potential solutions include restricting the task-level update (1b) to only a subset of parameters, or resorting to corresponding first-order variants such as FO-MAML.

Dear authors

I thank the authors for the effort in clarifying my concerns. I will change my score to weak accept.

Thank you for the interest in this work and the constructive feedback provided, which have been carefully addressed as follows.

Response to weaknesses

Due to limited time and computational resources, we are unfortunately unable to provide the results on the full MetaDataset that contains 240GB data. However, it is feasible to conduct a similar experiment on a smaller scale.

Before providing the details, we would like to emphasize that MetaDataset aims to assess the cross-domain generalization performance of meta-learning algorithms, which comprises 10 datasets from various domains, including ImageNet, Aircraft, and CUB. The standard setup involves meta-training a model on ImageNet (or the entire MetaDataset), and then meta-testing the trained model on all datasets.

Following the cross-domain few-shot learning setup in [1], we meta-train our prior model on miniImageNet, and meta-test it on tieredImageNet, Cars, and CUB datasets. As miniImageNet is a subset of the full ImageNet, we believe this test to some extent reflects the promising cross-domain generalization performance of our proposed algorithm. As shown in the Table below, our method consistently outperforms popular meta-learning approaches in such a setup, especially in the 1-shot case. This not only confirms the cross-domain generalization of MetaNCoV, but again justifies the importance of expressive prior when data are exceedingly limited.

[1] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals, "Rapid learning or feature reuse? towards understanding the effectiveness of MAML," ICLR 2020.

[2] C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," ICML 2017.

[3] J. Oh, H. Yoo, C. Kim, and S.-Y. Yun, "Boil: Towards representation change for few-shot learning," ICLR 2021.

[4] J. von Oswald, D. Zhao, S. Kobayashi, S. Schug, M. Caccia, N. Zucchet, and J. a. Sacramento, "Learning where to learn: Gradient sparsity in meta and continual learning," NeurIPS 2021.

[5] S. Kang, D. Hwang, M. Eo, T. Kim, and W. Rhee, "Meta-learning with a geometry-adaptive preconditioner," CVPR, 2023.

Response to questions

While our idea has the potential to be broadened beyond meta-learning, we must emphasize that our current setup is specifically tailored to meta-learning, which does not require a tractable pdf, but rather demands enhanced prior expressiveness. We should also highlight to the reviewer that the intractability of pdf prohibits learning NCoV via conventional approaches such as maximum likelihood training and evidence lower-bound maximization -- this thus necessitates careful attention and extra certain designs when applying our idea to other domains.