Thank you for the review. On ImageNet 64x64, the additional memory overhead of using LPIPS is less than 5%. We expect this would be even smaller on more large-scale settings since the generative model’s size would be relatively huge compared to the feature extractor such as AlexNet or VGG.

[1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

[2] Elucidating the Design Space of Diffusion-Based Generative Models

[3] Minimizing Trajectory Curvature of ODE-based Generative Models

[4] Scaling rectified flow transformers for high-resolution image synthesis

[5] Consistency Models

We appreciate the reviewer's valuable comments.

Why does the training loss of rectified flows (and FM with independent coupling) have a nonzero lower bound?

When two interpolation trajectories cross, given the intersection, there are two possible directions to go. As the neural net we use is deterministic, if there are two possible regression targets, the loss cannot be zero.

How did you decompose the loss? What is $\bar {\mathcal L}$?

As we described in the general response above, the L2 squared loss is lower-bounded by $\frac{1}{t^2} \mathbb E [|| \mathbf x - \mathbb E[\mathbf x | \mathbf x_t]||_2^2]$. $\bar {\mathcal L}$ is simply defined as the difference between $\mathcal L$ and its theoretical lower bound (i.e., $\bar {\mathcal L}:= \mathcal L - \frac{1}{t^2} \mathbb E [|| \mathbf x - \mathbb E[\mathbf x | \mathbf x_t]||_2^2]$), which is the only term we can reduce. The decomposition in Eq. 4 holds by definition.

“any other premetric .. deviates from the posterior expectation” — Why?

See general response.

Proposed techniques are ad-hoc, exist in the literature. “Focusing on the tasks where training loss is high” is a known technique.

“Putting emphasis on tasks with high training loss” does not always help; knowing how to apply it requires care.

E.g., one cannot naively apply this technique to 1-rectified flow training. Figure 1 in the attached PDF shows that the training loss of 1-rectified flow is also U-shaped as 2-rectified flow loss. However, the current state-of-the-art 1-rectified flow uses logit-normal distribution [4], which has the opposite shape.

On the other hand, for 2-rectified flow, we can design $p_t$ based on the loss shape as Sec. 3 tells us that the lower bound of the loss is nearly zero.

Sec. 3: If you are providing a guarantee, you must provide better formal proofs.

We are not making a formal statement—we do not expect our intuition to hold for every dataset/model. We found empirically that for many natural image datasets, our intuition appears to hold, allowing us to design significant training improvements.

Making section 3 formal poses several challenges. For example, we reason that if we add the difference of two synthetic data points to a noise, it is not a common noise realization that is used in the training. To make this formal, we would need to add restrictive assumptions to the data distribution, model, and training algorithm, as we are dealing with synthetic data. We also assumed that the model would not generate high-quality data if the input is very different from the noise it has seen during training, which is true in practical cases but also requires additional assumptions. As such, we ultimately decided to leave the intuition at an empirical level; we felt that adding a stylized theoretical result would not significantly strengthen the main results.

Q3: use a specific non-linear interpolation” — Isn’t the interpolation linear?

The variance-preserving diffusion model is a special case of rectified flows with the interpolation defined as $\mathbf x_t = \alpha(t) \mathbf x + \sqrt{1-\alpha(t)} \mathbf z$, where $\alpha(t) = \exp(-\frac{1}{2} \int_0^t (19.9s + 0.1)ds)$ (See Liu et al. [1]).

Q6:

What does “curvature” mean ?

When there is no intersection, the resulting (optimal) rectified flow's ODE trajectories are completely straight, or in other words, have zero curvature (Theorem 3.6 in Liu et al. [1]). Curvature [2,3] represents the degree to which the trajectory deviates from a straight path–one widely used definition is $\int_0^1 \|\mathbf{z}_1-\mathbf{z}_0-\frac{\partial}{\partial t} \mathbf{z}_t\|_2^2 dt$.

What equation needs to be proved to show that no trajectories from 1-RF coupling cross each other?

To show that there is no intersection, one needs to show that $\mathbb E[\mathbf x | \mathbf x_t = (1-t)\mathbf x' + t\mathbf z'] = \mathbf x'$ for all pairs of $\mathbf x'$ and $\mathbf z'$ and for all $t$.

Table.1:“(A) + something” — does that mean they are 3-RF (meaning 2 reflows) ?

All of our models are 2-rectified flows. “(A) + something” means we add the technique something to the base model configuration (A). We will clarify.

Sections 5.2 and 5.3 seem unnecessary.

We aim to draw attention to the potential of rectified flows. The computational cost to generate synthetic pairs is sometimes cited [5] as a downside, and we provide Section 5.2 as a counterargument. We introduced the new sampler in Section 5.3 to show that 2-rectified flow++ can be further improved by designing a better sampler–another advantage over other methods.

I doubt correctness of 5.3. Designing a solver require a lot more.

We did not mean to imply that our proposed update rule is itself an ODE solver, and will clarify. We use terms like “sampler” or “update rule” that are applied to existing solvers like Euler and Heun, instead of the term “solver” to refer to the new sampling algorithm. We do not even believe it converges to the true ODE solution in the limit as its behavior is very different from Euler solver’s in a large NFE setting.

Our claim is that the new update rule achieves better FID with a smaller NFE (Fig. 4).

Why are FIDs going up (fig.4) with higher NFE ?

The new sampler's FID goes up after some NFEs because it is not a proper solver, but its performance at low NFEs is better than the original sampler's best FID with many more NFEs (Figure 4).

Fig.5(b): The inverted noise norm distribution still looks quite different from the true noise... Not necessarily a good case for your method.

The inverted noise norm distribution is not the same as the true noise distribution but closer to it than the baseline. Also, the norm distribution is meant to complement the visual comparison in Figure 5(c), where we see that the inverted noises of the baseline exhibit a strong spatial correlation.

Fig. 2 caption: where is the factor $\frac{1-t}{t}$ ? Did you assume $t=0.5$?

Yes.

We appreciate the reviewer's valuable comments.

The main weakness of this paper is that all the improvements are incremental and empirical.

We respectfully disagree with the reviewer's comment. Our proposed techniques improve the FID of 2-rectified flow from 12.21 to 3.38 on CIFAR-10, 12.39 to 4.11 on AFHQ64, 8.84 to 5.21 on FFHQ64 compared to baseline, which is difficult to call incremental in any sense. Regarding our improvements being empirical, it is true that the paper is empirical at its core, motivated by the insights we provide in Sections 3 and 4. However, we disagree that this is a reason to reject. Indeed, many highly impactful papers in the area of generative modeling are empirical [3,4,5,6]. While we agree that having rigorous theoretical backing is more beneficial than not, we don’t believe it is always necessary or possible for all practical scenarios, especially those in deep learning.

In addition, many of the improved techniques overlap with existing diffusion improvements, such as Pseudo-Huber loss and Initialization with pre-trained diffusion models

Pseudo-Huber loss

The fact that Pseudo-Huber loss works well for consistency models does not necessarily mean that it should work well for rectified flows because of their fundamental differences in formulation. Consistency models' loss functions can be generalized to any metric function while rectified flow requires l2 squared loss (see the general response above). Otherwise, the obtained solution is not a posterior expectation and thus violates the marginal-preserving property of Liu et al. [1] (Theorem 3.3). Without our finding in Section 3, this generalization is not justified. Indeed, the 1-rectified flow’s FID gets worse if we use other losses, as we demonstrated in the supplementary results in the general response.

Pre-trained diffusion models

It is true that the Instaflow paper used the pre-trained diffusion models for 1) paired data generation and 2) initialization, but they did not provide any theoretical justification for why we can do so. This raises the question of whether doing so yields the same results as standard 2-rectified flow training since diffusion models are trained with a curved, nonlinear interpolation (see Figure 5 in Liu et al. [1]). An illustrative example is when we use the pre-trained variance-exploding EDMs. In this case, it doesn’t make sense to directly use them as they are trained in t \in [0, +\infty) while rectified flows operate on [0, 1] range, so we can intuitively see that some sort of conversion is needed.

In fact, it turns out that one needs to convert the scale and time following Proposition 1 in our paper to make them compatible; the proof is provided in Appendix D. Proposition 1 makes sure that 1) the coupling generated from a pre-trained diffusion model is the same as 1-rectified flow coupling required by Reflow algorithm (Algorithm 1), and 2) 2-rectified flow is properly initialized by 1-rectified flow model as suggested by Liu et al. [1]. Table 1 in the attached pdf file shows that naively initializing with EDM without the proposed conversion leads to slower convergence.

This result is quite surprising, as it suggests that 1-rectified flow (with linear interpolation) and diffusion models (with nonlinear interpolation) not only belong to the same model class but are actually equivalent (i.e., they are interchangeable after training by simple time and scale conversion). We believe that this is not widely known in the literature, as significant community effort has been invested in developing new training techniques for 1-rectified flows/flow matching models, with the hope that the linear interpolation “connects data and noise on a straight line” and thus “has better theoretical properties” [2].

The comparison in Section 5.2 may be somewhat unfair. The number of forward passes should also be considered to generate noise-sample pairs.

It already does so. The caption of Table 5 says, "Reflow uses 395M forward passes for generating pairs and 1, 433.6M for training.". They sum up to 1,828.6M.

[1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

[2] Scaling rectified flow transformers for high-resolution image synthesis

[3] Diffusion Models Beat GANs on Image Synthesis

[4] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

[5] Analyzing and Improving the Image Quality of StyleGAN

[6] Improved Techniques for Training GANs

We are sorry to bother you, but we wanted to check if you had further suggestions as the discussion period is ending today.

We appreciate the reviewer's valuable comments.

Have the authors tested with more than two-NFE like 100 to directly compare with 2-rectified flow? Does model still get consistent improvement?

We tested up to 16 NFEs and observed consistent improvement (see Figure 4). Compared to 2-rectified flow, 2-rectified flow++ with 2 NFEs outperforms 2-rectified flow with 110 NFEs by a large margin (2.76 vs 3.36, Table 2).

What is the advantage of U-shape time distribution over lognormal distribution? The intuition of this part does not clearly present.

Lognormal emphasizes the middle, while U-shaped distribution emphasizes both ends of the interval. The intuition is that when training 1-rectified flow (or diffusion model), there is little to learn on both ends because when t=1, a model just learns the dataset average, and when t=0, it just predicts noise average (i.e., 0). In contrast, in 2-rectified flow, a model should learn to generate a very sharp sample at t=1, and a very accurate noise at t=0. Therefore, the most difficult parts are the both ends.

Have the authors tested LPIPS-Huber-1/t (config F) for 1-rectified flow?

We emphasize that our scope is to improve the performance of the 2-rectified flow. 1-rectified flow training is not intersection-free, so using loss functions other than the squared L2 violates the theory. Nonetheless, we tested LPIPS-Huber-1/t loss for 1-rectified flow and provide the result in the general response. We see that the result is pretty bad. These results confirm our unique insights in 2-rectified flow presented in Section 3.

New sampler seems to have counter effect when many-NFE (>6) is used. Could the authors explain this behaviour?

The goal here is to obtain as good FID as possible using as few NFE as possible. The new sampler's FID goes up after some NFE but its performance with a low NFE is better than the original sampler's best FID with many more NFEs (see Figure 4). We introduced the new sampler as a prototypical example to show that our 2-rectified flow++ can be further improved by designing a better sampler, which is another advantage over other distillation methods. It is certainly possible to design a better sampler that does not have this increasing FID behavior, and we leave this to future work.

2-rectified flow++ falls behinds CTM and iCT.

Table 2 shows that our method outperforms CTM without GAN loss on CIFAR-10 by a large margin. 2-rectified flow++ can also be trained with GAN loss and may obtain a similar gain. Combining two different generative models is an interesting but orthogonal direction to our paper’s scope.

It is true that our 2-rectified flow++ falls behind iCT (FIDs 4.32 vs 4.01 on ImageNet64), but we want to emphasize that 2-rectified flow++ does not require real data during training while iCT does. These days, many foundation models such as SDXL [2] only disclose model weights but not training data. iCT cannot be trained in those cases.

[1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

[2] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

I appreciate the authors’ effort in conducting this experiment! It is interesting to see the combination of real and synthetic data could further minimize the learning gap between them and thus improve the performance. As authors mentioned, this has led to many potential rooms for exploration. I am looking forward to see this in your future work. So far, I am satisfied with the rebuttal and there are no major concerns left in other reviews, I raise my score to reflect this. Besides, there are some good resources for reference:

[1] Fan, Lijie, et al. "Scaling laws of synthetic images for model training... for now." In CVPR. 2024.

[2] Singh, Krishnakant, et al. "Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images." CVPR 2024 Workshop. 2024.