Q: Motivation and difference to flow matching

A: We’d like to clarify that the exact notion of Normalizing Flow (NF) we consider is here fundamentally different from the modern notion Flow Matching (FM) method. In our paper, we follow the conventional notation of NFs that exclusively refer to a maximum likelihood method that uses the change of variable technique of probability [1, 2, 3]. One of their distinct properties is that the training loss can be exactly computed for each training example. Flow matching [4] on the other hand, denotes a training technique that can be considered a variant of diffusion models, see [5] for such illustration, which does not follow a maximum likelihood objective. Also, the training loss of FM relies on a stochastic approximation of an expectation over time steps, and over the noise within each step. Moreover, it was shown that FM’s training objective follows a modified variational lower bound of likelihood [8], which again is fundamentally different from NFs's exact likelihood nature. In fact, this difference is also documented in [4], see the first paragraph of Section 5. The confusion may stem from the reference to continuous normalizing flow (CNF) in [4], which considers a generalized notion of NFs which deterministically map noise to data, regardless their training objectives. However, please note that the CNF here is mathematically equivalent to the probability flow [7], or ODE/DDIM style inference path in diffusion models [6]. This is also the reason why we have categorized all diffusion model and flow matching results under the term Diff/FM in our comparisons, see eg Table 2.

We hope this clarifies the difference, and we will be happy to make this more explicit in our paper, constraining the scope of our discussion of NFs to methods that directly follow the MLE objective with the change of variable formula.

Q: Label conditioning implementation

A: We adopt a simple implementation of label conditioning, where we add the label embedding directly to the position embeddings in each flow block. During guidance, the unconditional predictions are obtained by averaging the label embeddings of all classes and adding it to the position embeddings.

Q: Introduction to BPD

A: Thank you for the suggestion, we are happy to provide more context on it. In simple terms, BPD measures the average log probability for all the pixels of an image, where the probability is computed as a discrete one among 256 possible pixel values.

Q: Results worse than diffusion models

A: We agree that the current results of TarFlow is still lacking behind the best tuned diffusion models. We argue that for any new type of method, it usually takes time for it to be collectively improved by a community before it reaches the SoTA performance, and this was definitely the case for diffusion models too. We look forward to working with the generative modeling community together to further improve the upper bound of normalizing flow methods.

Q: Comparison with other flow and autoregressive model

A: Our comparisons have been focusing on mainstream continuous modeling method, and we have identified GANs, diffusion/flow matching models as the representative categories. Also note that we did not include traditional NF baselines in the FID comparisons as they were generally under performing and we were not able find comparable FID results reported in the literature. Following the reviewer's suggestion, we will also include representative baselines from Autoregressive models. Also, we would be happy to include more baselines for comparison if the reviewer is able to provide more specific references.

Q: Figure 4 not clear

A: We apologize for the visualization clarity, but the main cause for it is that the noise we apply is indeed very small and it does require a closer look to recognize the effect of denoising. We refer the review to more visualizations in the supplementary material, ie Figure 9-12, which should allow one to observe the effect of noise more clearly.

References

[1] Density estimation by dual ascent of the log-likelihood, Tabak et al, Communications in Mathematical Sciences, 2010

[2] Variational inference with normalizing flows, Rezende & Mohamed, ICML 2015

[3] Nice: Non-linear independent components estimation, Dinh et al, ICLR 2014

[4] Flow Matching for Generative Modeling, Lipman et al, ICLR 2023

[5] Diffusion Meets Flow Matching: Two Sides of the Same Coin, Gao et al, ICLR Blogposts 2025

[6] Denoising Diffusion Implicit Models, Song et al, ICLR 2021

[7] Score-Based Generative Modeling through Stochastic Differential Equations, Song et al, ICLR 2021

[8] Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation, Kingma & Gao, NeurIPS 2023

We thank the reviewer for acknowledging our contributions and we agree most of your assessments.

Q: Limitations of likelihood-based models

A: An interesting point of discussion the reviewer brought up is the fundamental limitations of likelihood-based models. We agree with the reviewer that likelihood method, especially density estimation such as NFs, can be ill behaved on real data that’s distributed on a narrow manifold. And we also agree that TarFlow needing input noise resonates with this view, which put in other words, suggests that density estimation is indeed a different task than generative modeling. We also believe that this observation is consistent with the findings in diffusion models, whereas the reweighting the likelihood based loss function is beneficial for sampling quality [1, 2].

Q: Significance of the BPD benchmark

A: A related question is whether one should value the strong BPD results that TarFlow achieves. We believe so, based on a subtle yet interesting point detailed below. Note that the BPD metric evaluates the discrete probability of quantized pixels, rather than the continuous density of real pixel values. This is a key difference because, unlike continuous density, discrete probability can always be faithfully evaluated even when the input has much lower intrinsic dimension (eg, concatenating a constant dimension to a discrete variable multiplies the joint discrete probability by 1, instead of infinity). When using a density estimation method to evaluate BPD, one would first inflate the discrete values $\tilde{x}$ to a local hyper cube $C(\tilde{x})$ that is disjoint from other discrete points (which is the case for the dequantization uniform noise), and convert the density model to discrete probability via integration, as in $\tilde{p}(\tilde{x}) = \int_{x \in C(\tilde{x})} p_{model}(x)dx$ (this is alo explained in the first paragraph of Section 2.4). Due to the same reasoning, the discrete probability result (hence BPD) is comparable among difference families of models, including discrete modeling methods using autoregressive models and continuous ones such as diffusion and NFs. In addition, the BPD metric has a concrete grounding itself, which essentially translates to the theoretical lower bounds for lossless compression of the target distribution [3], which is another way to prove that BPD does not degenerate. Therefore, we believe that TarFlow achieving SoTA BPD among different modeling methods is a strong indication of its raw modeling capacity.

Q: temperature based guidance for LLMs

A: We'd like first to confirm that we did come up with the attention based temperature guidance on our own, and we are not aware of a similar idea being deployed in other context such as LLMs. Like the reviewer, we are also intrigued by its generality and we believe that it does have potential to be applied to all Transformer based generative models as well. For LLMs, specifically, we speculate that guidance had received relatively little exploration likely due to the prevalence of finetuning, and it would indeed be interesting future work to further investigate its compatibility with our temperature based guidance.

References

[1] Denoising Diffusion Probabilistic Models, Ho et al, NeurIPS 2020

[2] Variational Diffusion Models, Kingma et al, NeurIPS 2021

[3] IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression, Berg et al, ICLR 2021

We thank the reviewer for the detailed review, and please see our responses below.

Q: Scalability claim

A: Our claim about the scalability of TarFlow is within the context of different model sizes and training FLOPs on a given dataset, which is supported by evidence in Sec 3.5 and Figure 6. Moreover, for the comparison of ImageNet 128x128 vs 64x64, we do not agree that the former is worse. Note that the absolute FID numbers are not comparable across datasets, but it is pretty clear that our ImageNet 128x128 samples are visually better than those on 64x64 (Figure 11 vs Figure 9).

In addition, we have conducted further experiments on ImageNet 256x256 with a model of ~1.4B parameters, and we are able to achieve very competitive results, with a 50K sample FID of 4.00 on pixel space. This number again places TarFlow in a diffusion dominated regime, see the table below as a comparison.

Q: Denoising & Section 2.5

A: We believe that the reviewer has a serious misunderstanding of our contributions. To clarify, denoising is performed with the same TarFlow model we train, and we do not use a separate denoising network. The exact recipe is explained in Equation 8, where $\log p_{model}(y)$ corresponds to the negative training loss of the TarFlow model. This also explains why Section 2.5 is a critical piece of our method, as it is not previously clear that NF models empirically give rise to accurate score estimates, making it a suitable choice for denoising in line with the Tweedie’s formula. Note that this point is also correctly recognized and acknowledged by Reviewer EWAZ as a strength of our paper.

As for speed, the denoising step adds a minimum overhead, due to its parallel nature. For actual measurements, we benchmarked the model we used for ablations, ie, the one from Figure 4, it takes 13.5 seconds to generate a noisy batch of examples, and 0.14s to denoise them.

Q: Results without noise injection

A: We assume that the reviewer is interested in the dequantization uniform noise setting, which is a standard in the NF literature [1] — and also the simplest way to make modeling pixels a valid density estimation problem (see, eg, Sec 3.1 of [3] for a discussion on this). As noted in the first paragraph of Section 3.3, this setting has poor numerical stability which makes it difficult to compare with other settings fairly. Nonetheless, during the rebuttal period we managed to train such a model by using fp32 instead of our default bf16, and performed 50K FID evaluation by skipping many samples with NaNs. See results in the table below.

Q: Non-causal transformer

A: First, the causal architecture is a necessary component for implementing the AR flow. More specifically, it allows us to compute the determinant analytically, and also invert the transformation explicitly, due to the Jacobian of the causal transformation being lower triangular. Both conditions will break if one directly applies a non-causal Transformer. That being said, the closest baseline we can think of that’s using a non-causal Transformer is to follow the channel coupling design from [1]. We performed such an ablation, and the result is shown below.

Q: Normalizing Flows other than Real NVP

A: We have also trained a volume preserving version, by enforcing the logdet terms to be zero. See results below.

Summary of additional ablations, where the baseline experimental setting follows that of Figure 4. All variants have inferior results than the default TarFlow setting.

Q: Novelty

A: We believe that TarFlow has significant novelty compared to [2]. The similarity is that they both apply Transformers to MAF. However, it is important to note that [2] does not provide a scalable recipe for high dimensional inputs. In particular, [2] only considers models with a single flow, whereas TarFlow stacks multiple flows with alternating directions, and show that this can be trained well. As shown in Figure 6(b) of our paper, using one flow (T=1) significantly limits the model’s capacity and it becomes a degenerated model for images. The other obvious difference lies in our usage of noise augmented training, denoising and guidance. All these critical pieces are completely missing in [2].

References

[1] Density estimation using Real NVP, Dinh et al, ICLR 2017

[2] Transformer Neural Autoregressive Flows, Patacchiola et al., 2024

[3] Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design, Ho et al, ICML 2019

We thank the reviewer for acknowledging our contributions and we answer each of the questions below.

First of all, please note that we do have a supplementary material where we include more experimental settings, related work and results which we believe might be interesting to the reviewer.

Q: The sampling time is very slow compared to standard autoregressive models.

A: Given a sequence length N (ie, number of patches), number of AR flows T and the Transformer depth for each flow L, the inference cost of TarFlow evaluates to $O(TLN^2)$. This cost will be equivalent to that of an autoregressive model on the same sequence but with a depth of $TL$. Therefore, the inference speed comparison between TarFlow and an AR model boils down to the combined depth, and the two families of models actually have the same inference time complexity given the same depth budget.

Q: The architecture of model is quite heavy 8x8 = 64 transformer blocks, which is about 2 time larger than DiT XL

A: Related to the previous question, our design philosophy has been to train strong models and prove that NFs are capable learning objectives, whereas we did not emphasize on reducing the total depth. As for the comparison to DiT, we argue that they are also not directly comparable, due to two reasons. 1. DiT conducts experiments in the latent setting, which is known to simplify the modeling difficulty, whereas TarFlows are pixel based models. 2. The inference mode of DiTs is actually much deeper than TarFlow, due to the need of calling multiple NFEs.

Q: The method is not new and a little bit limited novelty since classifier free guidance and data augmentation are existed techniques

A: We respectfully disagree with the reviewer’s assessment on novelty. It is indeed true that all the essential ingredients in this work has been previously established techniques, this includes MAF, Transformers, noise augmentation, denoising and guidance. However, in the context of normalizing flows, the exact recipe of using them has been unknown, and as the reviewer already acknowledged, we are the first work to show the full potential of correctly combining them together. The fact that these individual techniques are standard only strengthens the simplicity aspect of our work, and it should not be treated as a penalty for novelty. In fact, the same argument can be made for many groundbreaking works — for example, denoising autoencoders are well studied subjects [1], but correctly applying it to generative modeling [2, 3] is still highly novel and it gives rise to the diffusion and score based generative model revolution.

Q: Is the anyway to develop faster inference model based on TarFlow ?

A: Although we did not focus on inference speed in this work, we are excited by several possibilities of improving it in the future work. First of all, there is a large body of literature (eg, speculative decoding [4]) dedicated to speeding up the inference of autoregressive models, which TarFlow can already benefit from, due to its autoregressive architecture. Another promising direction is distillation, which has achieved great success in diffusion models [5]. And we believe that distillation should be naturally compatible with TarFlow, and arguably more so than with diffusion models, as we have an explicit bijective mapping between an input x and noise z.

References

[1] Extracting and composing robust features with denoising autoencoders, Vincent et al, ICML 2008

[2] Denoising Diffusion Probabilistic Models, Ho et al, NeurIPS 2020

[3] Generative Modeling by Estimating Gradients of the Data Distribution, Song & Ermon, NeurIPS 2019

[4] Fast inference from transformers via speculative decoding, Leviathan et al, ICML 2023

[5] Progressive Distillation for Fast Sampling of Diffusion Models, Salimans & Ho, ICLR 2023