SCORE NEURAL OPERATOR: A GENERATIVE MODEL FOR LEARNING AND GENERALIZING ACROSS MULTIPLE PROBABILITY DISTRIBUTIONS

The paper is extremely poorly written. It makes it extra hard to understand. Definitions are unclear and change over the course of the paper. Its a guessing game to understand what is what. I had an horrible time trying to understand the paper. Readers who are not reviewers will clearly just skip this paper given how badly written it is.

The paper title and abstract needs refactoring to be comprehensible when reading for the first time. The paper needs to actually define and explains things properly.

Experiments are nice but very toy-ish, 2D Gaussians and pairs of MNIST digits. You need something like ImageNet with some classes never seen. CIFAR-100 also could be a start, remove like 5 classes from it for test time. This is whole idea of out-of-distribution generalization. There are existing benchmarks for that, but its more for classification. But you could just reuse them for generation.See Domainbed, https://github.com/facebookresearch/DomainBed, you train on some domains and not on others. This would be perfect to test your model. Also it has been shown that diffusion models can do zero-shot classification (https://arxiv.org/abs/2303.16203), you could even test your approach on the actual DomainBed challenge for classification using your score-model! Sky is the limit, but right now the paper is not ambitious enough and its not looking into interesting applications. With a bit more ambition, this could be a very interesting paper.

It needs more extensive literature review for meta-learning approaches.

The paper has a few weaknesses that could be addressed:

• Writing: In general, the writing could use some improvement. For example, citations are poorly done, with incorrect usage of \citep and \citet (L34-42,L59,L46-50, etc.) especially in places where the authors themselves write a paper’s author names, followed by \citet. (E.g., L86,89,91). These could be fixed. I see similar issues with equations (L129).

• Math: Sometimes, the notations are defined when they first appear, and this could cause confusion. For example, the definition of $\mathbf{u}$ in L210 is not accompanied by a definition of what it represents. Similarly, the use of $\mathbf{G}$ in Eqn. 14, or $\nu$ in Eqn 12. In most cases, this is described eventually, say half a page later, but this does not yield a good reading experience and could be fixed.

• Experimental validation: Though the proposed idea and methodology seems promising, the experiments, in my opinion, don’t live up to the claim. Unless I am misunderstanding, It would be best to either make the claims more tamer, or have experiments the validate the existing claim that the SNO would be able to work few-shot with any data distribution. The experiments on 2-digit MNIST with a hold out set do not seem convincing. Even if they chose to work with lower resolution images, it would be more appealing to see actual generalization across datasets. For example, how about using SVHN (greyscale) and MNIST. Both being digit datasets, the task would be manageably hard, but possible also able to justify that few-shot alignment to a brand new dataset is indeed possible.

Discrepancy Between Claims and Experimental Evidence:

• While the paper claims that SNO can improve performance across various distributions, the chosen datasets—GMM and MNIST—are actually toy dataset. These limited datasets are insufficient to convincingly demonstrate the model’s effectiveness for handling complex and high-dimensional distributions. If SNO’s strength lies in handling diverse and complex data, additional experiments with richer and more challenging datasets are necessary to substantiate this claim.

• The experiments are conducted in toy and small datasets such as Gaussian Mixture Models or 1024-dimensional MNIST. From my experience working with these datasets, it is hard to expect that findings in these datasets naturally generalize to real-world datasets.

• The idea of learning one generative model across different data distributions was originally discussed in "Distribution Augmentation for Generative Modeling" by Jun et al. ICML 2020. This paper shows that one can learn one generative across different distributions by providing a simple input conditioning that specifies the target distribution. This idea has been explored in diffusion modeling literature as well (for example see "Elucidating the Design Space of Diffusion-Based Generative Models" by Karras et al. NeurIP 2022). This submission does not discuss the idea of specifying the data distribution via simple embedding. It also does not discuss why a simple approach like passing a data distribution embedding would enable learning a diffusion model across different distributions.

• The paper is hard to follow. Section 3.2 suddenly introduces many different concepts without explaining how they fit into the bigger picture.

• Equation 18 trains the VAE and SGM prior jointly. However, the KL is evaluated with respect to a standard Normal prior in Eq. 16 while the prior is set to be a diffusion model.

1) Readability

Usually one doesn't mark the first weakness on the list to be readability, but in this caseI find it crucial. I was having a very hard time to read this paper and follow the derivations.

1. First the structure is hard to follow. I would expect starting from high level and then get in to low level derivations. I don't mean the intro as the high level, but for example in section 4, I would expect to see clearly what is the input, output and goal.
2. I think some simple examples are also needed for inputs and expected outputs.
3. A figure describing the method is also missing.
4. In the background I think a brief explanation about NOMAD is needed.
5. Taking Figure 1 as a test case: It took me unreasonable amount of time to understand it. The related text in sec 5.2 is confusing and almost cryptic. It is not clear what is Gaussian there, no motivation for this very specific choice of data. No visual explanation in the figure itself. The color coding is not clear to me. The caption says "the first two are derived from the training sets, and the last two are from the testing sets." does that mean the left two and right two? If so why not putting some lines making a table with titles? Also if that is the case, I don't understand why the rightmost column has examples that are "left col right row" which is a training characteristic, not test.

2) Comparing to conditional generative models

1. Simply defining the goal, I did not see any definition that makes the premise of this work actually different from a conditional generative model. Conditional generative models in many cases get conditions that they were never trained on. They produce samples from a conditional distribution that I think can be any one of the examples in the experiments. In general I think that explaining why the proposed method is different/better than conditional models should be the most important claim to justify in this paper.
2. The authors do use a baseline they refer to as a conditional model. To my understanding it is something else. A model that is fine-tuned on the unseen distributions. First, this is not what a conditional model is- a conditional model gets a condition along with its input. Moreover, comparing to fine-tuned models implemented by the authors is always problematic. We don't have any details about how many iterations, what was the procedure, what weights trained etc. Without pointing any finger to any.one, usually authors don't have a huge motivation to make a baseline work better and fine-tuning is very tricky.
3. Mathematically, while I appreciate the interesting derivation, as far as I see, the objective boils down to a standard conditional setup. eq.13 is eq.3 with an additional conditioned term, sampled from a distribution of conditions. $s_\theta(u^v,z(t),t)$. I think any condition can be framed as a representation of a distribution. If the novelty is in the specific representation, then the premise should have been around it and indicating differences from the representation used in NOMAD. If not, then the contribution of the paper is combining NOMAD with conditional score models.
4. l503: "conditional score-based generative model cannot generalize to unseen test distributions once trained." I don't agree with that statement. While it might be true for discrete class-conditioning, it is not true in most other cases. Conditioning can be done for example on images in various ways and of course can be applied with an unseen image. Text-to-image models are another example for conditioning that generalizes to not necessarily seen conditions.

3) Experiments

1. In general experiments are done on very low-scale examples and very few. While I wouldn't expect generating HD videos with a novel method, I do think that more evidence for its capabilities is needed.
2. The toy example experiment is very specific and complicated. Why not choosing simple distributions? Like spirals or just Gaussians, as used in many generative models works? Seeing such examples raises the concern whether they were carefully chosen.
3. I'm not sure about the few-shot learning part. I think the fact that it is possible to use a single example actually very strongly shows that this is equivalent to a conditional model. If it really adhered accurately to this distribution it should, in fact, make a Dirac Delta over the given example and generate only that. Thinking about it as a condition also makes this application less impressive, as single images as conditions are common.