Adversarial Self Flow Matching: Few-steps Image Generation with Straight Flows

We appreciate the reviewer's comments, and we provide clarifications on these issues below.

Q1: More generated images from other baselines.

A1: We provide 1-Rectified Flow, 2-Rectified Flow and 3-Rectified Flow generated images with NFE=1and NFE=2 on CIFAR-10 for comparison in Appendix G.

Q2: High-resolution images generated by ASFM with fewer steps.

A2: We provide ASFM generated images on CelebA-HQ and AFHQ-Cat with NFE=1 and NFE=2 in Appendix F and we also provide 1-Rectified Flow generated images on CelebA-HQ and AFHQ-Cat with NFE=1, NFE=2 and NFE=6 in Appendix G.

Q3: ASFM can generated straight flows.

A3: ASFM's ability to generate straight flows is directly evidenced by its capacity to produce high-quality generated images with very few sampling steps. This is because only sufficiently straight flows can yield accurate solutions with minimal steps when solving the ODE numerically. Additionally, ASFM's performance on the CIFAR-10 dataset, where it achieves results very close to those obtained with 113 steps using only two steps, further demonstrates its capability to generate straight flows, as shown in the following table.

Q4: The clarity of Figure 2.

A4: We have clarified the meaning of the transparent red dashed line in Figure 2, indicating that it represents the vector field trained with paired data $(x_0, x_1)$ and $(x_0', x_1')$ which have not been repaired. The Self Online Training process illustrated in Figure 2 can be understood as follows: 1) Initially, during the training of the model, if there are intersections in the connections between paired data$(x_0, x_1)$ and $(x_0', x_1')$, the learned vector field $f_t$ (the transparent red dashed line) will approach the intersection point. 2) Due to the non-crossing nature of flow fields, the learned vector field will generate data that repairs the previously paired data, resulting in repaired data $(x_0, x_1')$ and $(x_0', x_1)$ . 3)The connections of the repaired data no longer have intersections, and the newly learned vector field $f_t'$ (the opaque red dashed line) will become straighter.

Q5: The clarity of mixture experiment.

A5: The mixture experiment is designed to demonstrate ASFM's zero-shot image editing ability, meaning its capability to perform image editing without additional training. In the mixture experiment, we selected two images of cats. Simply stitching these two images together results in an unnatural outcome. However, using ASFM, we can obtain the latent space representations of the two images (by solving the ODE backward to find their corresponding points in the initial distribution). By stitching these two latent space representations together, adding some noise, and passing them through ASFM, we can generate a more natural mixed image. This mixed image retains features from both original cat images, such as a yellow upper half and a white lower half.

Q5: On high-resolution datasets, the FID of images generated by ASFM with NFE=1 is greater than 10.0.

A5: We are not sure about the basis for the standard that the image quality with NFE=1 should achieve an FID below 10.0. If possible, we kindly ask the reviewer to provide the source for this standard. Furthermore, we believe that, as shown in Table 2, ASFM has already demonstrated outstanding results compared to other methods.

Q6: ASFM does not use "original flow model".

A6: We believe that the naming convention of Rectified Flow might have caused some confusion for the reviewer. The "1-rectified flow" could be what the reviewer referred to as the "origin flow model," as 1-rectified flow does not use the Rectify operation. Only when $n \geq 2$ does the n-rectified flow employ the Rectify operation.

Q7: Computation of online sampling and using same t in Algorithm 1.

A7: Compared to the Rectify operation proposed in Rectified Flow, ASFM's online sampling incurs lower computational overhead. For instance, as mentioned in the official implementation on Rectified Flow's GitHub repository, they recommend generating at least $1,000,000$ pairs of data for reflow on the CIFAR-10 dataset. In contrast, ASFM requires at most $(1600+127)×512=884,224$ pairs of data on CIFAR-10.

Regarding the use of a random $t$ for the two interpolations in Algorithm 1, we believe that as training progresses, using the same $t$ or different $t$ values will sufficiently sample $t$, so the impact on training should be minimal. Nevertheless, this is an approach worth exploring. However, due to time constraints, we leave this for future work.

Q8: The size of Training Memory.

A8: For the selection of the hyperparameter Training Memory Size $k$ (batches), we aim for the number of images in the Training Memory to slightly exceed the corresponding dataset size. For instance, on the CIFAR-10 dataset, the Training Memory contains $512 \times 128 = 65,536 \simeq 50,000 + 10,000$ (the size of CIFAR-10 train + CIFAR-10 test) images. Similarly, on the AFHQ-Cat dataset, the Training Memory contains $64 \times 128 = 8,192 \simeq 5,653$ (the size of AFHQ-Cat) images. However, due to the higher resolution and the large size of the CelebA-HQ dataset, which contains 30,000 images, generating excessive data can impose storage pressure. Therefore, we choose the Training Memory Size $k$ (batches) for CelebA-HQ to be consistent with that of AFHQ-Cat.

Q9: Ablation study of $\lambda_1$ and $\lambda_2$.

A9: Thank the reviewer for this comment. We have added the relevant ablation experiments in Appendix E.

We sincerely appreciate the reviewer for the constructive and insightful comments, and we will provide detailed responses to these issues below.

Q1: Adversarial training in one-step generation has already been used in some works.

A1: First, we would like to emphasize that our goal is not to achieve one-step generation but rather to utilize adversarial training to narrow the gap between the generated data distribution and the real data distribution. By combining this with Online Self Training, we aim to develop ODE-based generative models capable of generating straight flows, as indicated in the paper's title. Second, the references provided by the reviewer are fundamentally different from ASFM. CTM is based on consistency models; ADD is a distillation-based method; and ASM introduces adversarial training into score matching. The methods proposed in these works differ significantly from Flow Matching. To the best of our knowledge, ASFM is the first to apply adversarial training to Flow Matching in image generation, achieving highly competitive results among various Flow Matching-based methods.

Q2: The effectiveness of ASFM.

A2: Under the same number of NFEs, ASFM achieves significantly better generation performance on CelebA-HQ and AFHQ-Cat compared to other Flow Matching-based methods. As shown in Table 2, with NFE=6, ASFM reduces the FID on the AFHQ-Cat dataset by at least 34% compared to other methods (ASFM vs. BOSS) and reduces the FID on the CelebA-HQ dataset by at least 56% (ASFM vs. BOSS). This directly demonstrates the effectiveness of ASFM on high-resolution datasets.

ASFM's one-step generation results on high-resolution datasets outperform the results of 1-Rectified Flow with six steps, and ASFM's two-step generation results are either superior to or comparable with Consistency-FM's six-step results. This highlights the superior image quality generated by ASFM under extremely few sampling steps. Rectified Flow did not conduct experiments with 2-rectified flow or 3-rectified flow on high-resolution datasets such as CelebA-HQ (256x256) and AFHQ-Cat (256x256). Thus, we are unclear about the reviewer's statement that Rectified Flow has been "shown to achieve better FID scores under few-steps sampling."

On the CIFAR-10 dataset, with NFE=2, ASFM demonstrates significantly better generation performance than 2-Rectified Flow and 3-Rectified Flow, as shown in Table 1. Moreover, on CIFAR-10, the distilled results of 1-Rectified Flow (6.18), 2-Rectified Flow (4.85), and 3-Rectified Flow (5.21) are very close to ASFM's generation result with NFE=1 (5.07), although we believe comparing ASFM with distillation-based methods is not entirely fair.

Q3: Update Table 2 with some works unrelated to Flow Matching.

A3: ASFM is consistently based on Flow Matching, offering advantages not found in other methods such as CTM. For example, ASFM is theoretically grounded in ODE-based generative models. Due to the reversible nature of ODEs, ASFM can identify the latent space representation of any image within the initial distribution, facilitating tasks like image interpolation and zero-shot image editing, as demonstrated in Appendices C and D. In contrast, methods based on consistency models, such as CTM, lack an explicit ODE solver, complicating the inversion process. Given the fundamental theoretical differences between these methods and ASFM, comparing ASFM with them holds significant value.

Q4: Add distilled rectified flow results in Table1.

A4: Same as our response A2, the distilled results of 1-Rectified Flow (6.18), 2-Rectified Flow (4.85), and 3-Rectified Flow (5.21) are very close to ASFM's generation result with NFE=1 (5.07), although we believe comparing ASFM with distillation-based methods is not entirely fair.

Thank you for addressing my questions. However, I am not fully satisfied with the authors’ rebuttal.

Q1: Adversarial training in one-step generation has already been used in some works.

The explanation is not convincing to me. While the goal of the proposed method may extend beyond one-step generation, its primary focus remains on minimizing the gap between real and generated data in fewer-step settings. Therefore, comparisons with other one-step and few-step generation methods (e.g. CTM and CD) are valid and important, not necessarily limited to flow-matching methods. So, the argument "Q3: Update Table 2 with some works unrelated to Flow Matching." does not make sense at all. Furthermore, the authors overstate their contribution by claiming to be the first to use adversarial training for flow matching. It is important to note that flow matching and score-based methods have been proven equivalent in [1], which shows minimal differences between the proposed method and the ASM paper.

Q2: The effectiveness of ASFM.

For CIFAR10, the argument "Moreover, on CIFAR-10, the distilled results of 1-Rectified Flow (6.18), 2-Rectified Flow (4.85), and 3-Rectified Flow (5.21) are very close to ASFM's generation result with NFE=1 (5.07), although we believe comparing ASFM with distillation-based methods is not entirely fair." is not valid. Since, your method can be considered as a different type of flow straightening, the proposed method still benefits from a further distillation stage. So, the comparisons with distillation-based models are strongly necessary.

For high-res data, the authors’ arguments for neglecting the results of RectifiedFlow on Celeb and AFHQ-Cat are not valid. As RectifiedFlow is a key baseline for the proposed method, the authors should provide these missing results (even if they were not originally included in the baseline) to ensure a fair comparison.

Regarding the statement, “shown to achieve better FID scores under few-steps sampling,” my observation on CIFAR-10 indicates that multi-step rectification yields better model performance under the same NFE settings. Therefore, I believe this observation also holds for high-resolution datasets like Celeb and AFHQ-Cat, as supported by [2, 3].

Q4: Add distilled rectified flow results in Table1.

Once again, this excuse is not valid. Please refer to my concerns above.

[1] Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation, https://arxiv.org/abs/2303.00848 [2] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, https://arxiv.org/abs/2403.03206 [3] InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, https://arxiv.org/abs/2309.06380v2

We would like to sincerely thank the reviewer for the thoughtful and valuable comments on our work. We will address these issues and provide a detailed explanation accordingly, as shown below.

Q1: Explanation of the straightening effect of Self Online Training.

A1: In fact, both the triangle inequality and the shortest distance between two points being along the straight line hold in any finite-dimensional Euclidean space. In simple terms, when the line connecting two pairs in the training data intersects at a certain point, the vector field learned by the neural network will pass through the vicinity of the intersection and "repair" the two pairs. The repaired line segments will no longer intersect, resulting in a vector field that is straighter.

Q2: Regarding Model A's failure to converge.

A2: When we conducted the experiments, we found that directly using Online Self Training, with the Training Memory size set to 1, caused the model to fail to converge, as shown by the yellow lines in Figure 6 (a) and (b). Despite the use of the GAN loss to help align the generated data distribution more closely with the real data distribution, we believe that due to the online training nature of ASFM, training with a small Training Memory leads to overfitting on the data in the Training Memory before the memory is updated. This, in turn, degrades the quality of the next batch of generated data in the online training process, causing the model to fail to converge. This is why, when designing the Training Memory size for ASFM on the CIFAR-10 dataset, we chose a memory size that is comparable to the size of the CIFAR-10 training set.

Q3: Initialization of the network

A3: Even with randomly initialized model parameters, we believe that ASFM can generate data that aligns with the real data distribution, as the discriminator provides information about the real data distribution to the model. However, the random initialization of model parameters presents challenges for data generation during training. Methods and parameters such as random initialization and the weight of the GAN loss need to be explored experimentally. Due to time constraints, we leave this as a direction for future research.

Q4: Ablation study of parameter freq.

A4: We included ablation studies on hyperparameters such as freq. Please refer to Appendix E, Figure 10, which demonstrates the effectiveness of our hyperparameter choices.

Q5: GAN loss may lead to unstable training.

A5: During the training process of ASFM, the FID gradually decreases as the number of training steps increases, demonstrating the stability of the training process, as shown in Figure 6. This may be because, during training, ASFM does not directly input the final sampled results into the discriminator. Instead, it uses the data generated in one step as a correlated representation of the final sampling result to input into the discriminator, thereby reducing the impact of the GAN loss on the training process.

Q6: If ASFM suffer from model collapse?

A6: We computed the recall metrics of ASFM on the CIFAR10 dataset with NFE=1 and NFE=2, and compared them with methods such as 1-rectified flow, as shown in the following table. The results show that ASFM achieves the best recall metrics, which we attribute to the following factors: 1) The training data is generated online, which ensures more comprehensive sampling of the distribution, thereby avoiding situations where certain modes cannot be generated. 2) ASFM’s use of the GAN loss occurs during the one-step generation process within training, where this one-step generation serves as a representation rather than a method for generating data. As a result, ASFM's sensitivity to the GAN loss is limited.

Q7: The results of ASFM with more NFEs on CIFAR10.

A7: Since the flows produced by ASFM are sufficiently straight, results similar to those obtained with more sampling steps can be achieved with fewer steps, as the following table shows.

We would like to sincerely thank the reviewer for the positive and thoughtful comments on our work. We address these questions and provide a detailed explanation as described in the following.

Q1: The trade-off between image quality and NFE?

A1: Since the flows produced by ASFM are sufficiently straight, results similar to those obtained with more sampling steps can be achieved with fewer steps. For example, on the CIFAR-10 dataset, the generation results using the Euler solver with two sampling steps and the results using the RK45 solver are shown below. Although their NFEs (Number of Function Evaluations) differ significantly, the generation quality is quite similar.

In the case of a few sampling steps, increasing the number of steps can improve the generation quality. As shown in Table 1, when using the Euler solver in ASFM, increasing the sampling steps from 1 to 2 leads to an improvement in sampling performance (FID decreases and IS increases).

Q2: Generalization to other tasks.

A2: Extending ASFM to conditional generation tasks such as text-to-image generation, is a highly worthwhile research problem. We plan to address this issue in our future work.