Meta-Learning Universal Priors Using Non-Injective Normalizing Flows

• I am curious about the novelty of this paper in relation to Sylvester NF. This paper proposes the use of non-injective normalizing flows, specifically Sylvester NF (van den Berg et al., 2018). It would be beneficial to discuss the differences between the proposed method and the approach presented by van den Berg et al. in 2018.

• This paper introduces Sylvester NF for $f: z \in \mathbb{R}^d \mapsto z' \in \mathbb{R}^d$, where $d$ is the dimensionality of neural network parameters ($\phi_t \in \mathbb{R}^d$). This limitation could hinder the scalability of the proposed method when dealing with an increasing number of parameters in neural networks.

The overall workflow of the description on theorem formulation is a bit scattered. Work is needed to make the description on theorems more logical with each other. Also, please check if some of the math concepts are really necessary in the formulation.

In section 3.1, clarification is needed on why Pz'(z') defined based on the bijection function f is a good approximation of q.

Is some of the notions described in the article closely related to prior design of meta learning? e.g. what is the strength of pushforward measure? why are we choosing Borel sigma-algebra for \mathcal{B} in theorem 3.1 and not other algebras?

Given the significant improvement of performance on 1-shot few shot learning of miniimagenet, I would assume significantly richer information is encoded in this prior. How does this proposed prior hold this much additional information?

In section 3.3, why “optimization of the transformation parameter \theta_f” can be replaced with "optimize the latent vector z_t", given \theta_f and z_t are totally different in meaning.

minors:

What is “meaning fitting ability”? Explanation needed.

"when data are scarce or costly to annotate. Examples of such applications include machine translation", the translation task don't have this data scarcity problem.

"Conventional approaches to meta-learning have relied on hand-crafted techniques to extract prior knowledge" some recent works to support this.

The paper unfortunately has some weaknesses, some of them might be to my misunderstandings tho.

1. the literature review in the normalizing flow direction is not complete. Some of the paper deals with the tails of pdfs. A detailed account of NFs trying to model heavy (light)-tailed distributions in [1]. The paper furthermore ignores important recent literature on NFs which is very related, for instance [2], [3].

2. I am not sure about the novelty of theorem 3.1. It is presented and described in a way that lets one think this might be novel, but I think this is an easy consequence of the monge map being the gradient of a convex function. The Monge map should exist under these conditions, and as the gradient of a convex function it should also be what the authors call weakly increasing. (Also I think the authors should define weakly increasing as this is not standard defintions. It is only explained in the proof)

Or to see it differently look at the Explanation of [1], where they argue how to construct the well-known Knothe-Rosenblatt transform, which should also give the same results as theorem 3.1.

3. I dont understand non-injective Sylvester Flows. In the usual Sylvester Flow paper a more specific structure is used to ensure invertibility (they find conditions on the diagonal of the triangular matrices so that it is invertible). Here this looks more like a ResNet. Also the algorithm does not use any NF related concepts such as maximum likelihood training and only forward evals are needed. I am not sure whether this should even be called a normalizing flow? the only similarity i see is using a prior/latent distribution.

4. I find it quite weird that the validation data is also used when optimizing the Sylvester NF. This however might be very common practice in meta learning, which I am not familiar with.

5. The motivation for non injective flows in general is not super convincing to me. There have been some results (for specific NFs) that they are universal distribution approximators, i.e., having two lebesgue densities one can at least find a sequence of NFs with distance between the distributions going to 0, see e.g. [4].

[1] Tails of Lipschitz Triangular Flows, Jaini et al

[2] Generalization of the Change of Variables Formula with Applications to Residual Flows, Koenen et al

[3] MetaFlow: A Meta Learning With Normalizing Flows Approach For Few Shots Learning, Ahmed et al

[4] Coupling-based invertible neural networks are universal diffeomorphism approximators, Teshima et al