Taming Rectified Flow for Inversion and Editing

Dear Reviewer 5wXG,

Thank you for your comprehensive and detailed review of our paper and the recognition of our work's effectiveness. We provide our feedback as follows.

Experiment to demonstrate the effectiveness when increasing the order using Taylor expansion, which is the key idea of the paper.

Thanks for your advice! We have provided experiments on both image generation and reconstruction in Table5.

The results illustrate that under various timesteps, our methods outperform the baseline significantly. With the higher order expansion (RF-Solver-3), the performance becomes even better. Although the comparison is conducted with the same number of timesteps (rather than the same NFE), we notice that under similar NFE, a higher order expansion sometimes also illustrates a better performance. For example, considering the 20 timesteps for Vanilla RF, 10 timesteps for RF-Solver-2, and 7 timesteps for RF-Solver-3, the NFE for them is similar (20, 20, 21, respectively), while RF-Solver-3 illustrates the best performance among them.

Hyperparameter robustness for feature sharing

We have provided a more detailed analysis about the choice of feature sharing in Figure11, Figure12, Figure13, Figure14. In our work, solely tuning the hyperparameter of the feature-sharing step is enough to obtain a satisfying result. What's more, editing a high-resolution image (1360 * 768) using our methods only takes a short time (less than 1 minute). As a result, we believe that the parameter-tuning is acceptable for most users.

What happens if you only have RF-solver but not RF-edit for editing

As mentioned in Line 242~258 in the main paper, only using RF-Solver for editing sometimes cannot maintain the consistency between the source image and the target image. Figure 7 illustrates some results produced solely by RF-Solver. We also provide a quantitative ablation study about this in Table7.

What's the memory for storing Vs? Why not storing/sharing Ks or attention maps?

For editing a 1360*768 image, the total memory needed to store the feature is about 18G. We store the feature in the CPU, rather than GPU Memory. In practice, this would not significantly reduce the efficiency of inversion and editing.

We also add the qualitative results about sharing K or attention maps in Figure13. Experimental results demonstrate that, compared to K or the attention map, V contains the richest information regarding the original image. Sharing V can effectively preserve the source image's details in a relatively small number of feature-sharing steps, whereas sharing K and the attention map under the same number of steps yields less satisfactory outcomes. Therefore, we choose to share V for its effectiveness and efficiency.

Dear Reviewer 1xw9,

We sincerely thank for your recognition of the methods and experiments in our work!

I saw there is a hunyuan video editing code; is there any results on this?

We provided some video editing results produced by HunyuanVideo in Figure 9 in the Appendix. We also provide the code for video editing in the supplementary.

The quantitative results are shown in the table below, where the experimental settings follow Table 4 in the main paper. We will also add this to the main paper.

Table 9. Quantitative results about HunyuanVideo

Dear Reviewer eQrG,

Thanks for your comprehensive review and insightful comments on our paper. We appreciate that you recognize the motivation and performance of our methods. The response to your concerns is shown below.

Implementation Details about HunyuanVideo

Thanks for your advice! For video editing on HunyuanVideo, total timesteps are set to 25, and the feature-sharing step is between 1 to 5 for different cases. We will add these implementation details in the main paper. What's more, we have provided the code in the supplementary materials and planned to release it in the future.

The authors should provide qualitative and quantitative results to demonstrate the benefits of RF-Solver without feature-sharing.

Thanks for your advice, we have provided the qualitative results of feature-sharing in RF-Edit in Table7. The quantitative results are provided in Figure 7 in the paper.

What's more, we also provide a more detailed analysis about the feature sharing strategy in Figure11, Figure12, Figure13, Figure14. We finally choose to share the V feature in the last 20 single-stream blocks of FLUX in the timesteps near to noise, which is proved to be the most effective choice.

Qualitative comparisons between RF-Edit and previous methods such as MasaCtrl and PnP.

Thanks for your advice! The results are provided in Table6 and Figure15. The qualitative comparisons between our method and PnP are shown in Figure 5. Our method outperforms the baselines both qualitatively and quantitatively.

Additional ablation studies using the same number of timesteps

Thanks for your advice. We have provided more detailed experiments on both image generation tasks and reconstruction tasks in Table5.

The results illustrate that with the increase of Taylor Expansion order, our method illustrates a better performance at various timesteps. Although the comparison is conducted with the same number of timesteps (rather than the same NFE), we notice that under similar NFE, a higher order expansion sometimes also illustrates a better performance. For example, considering the 20 timesteps for Vanilla RF, 10 timesteps for RF-Solver-2, and 7 timesteps for RF-Solver-3, the NFE for them is similar (20, 20, 21, respectively), while RF-Solver-3 illustrates the best performance among them.

Dear Reviewer n1SC,

Thanks for your time and thoughtful review! We appreciate your recognition of the effectiveness of our methods. Here is our feedback:

More Explanations about Our Methods

RF-Solver

If we substitute the formulation of derivative into Equation 9, then the formulation becomes

$Z_{t_{i+1}} = Z_{t_{i}} + ({t_{i+1}} - {t_{i}}) v_\theta (Z_{t_{i}}, t_{i}) + \frac{1}{2} (t_{i+1} - t_{i})^2 \cdot \frac{v_\theta (Z_{t_{i} + \Delta t}, t_{i} + \Delta t) - v_\theta (Z_{t_{i}}, t_{i})}{\Delta t}.$

In the above formula, $\Delta t$ is required to be set to a sufficiently small value to estimate the derivative with less error (as specified in Line 187~188 in paper), which is not equivalent to $t_{i+1} - t_{i}$. As a result, RF-Solver exhibits a clear difference from the Heun method. RF-Solver also outperforms Heun method (Table 1).

What's more, RF-Solver is not limited to 2nd order expansion. We derive the general form (Equation 7) by firstly deriving the exact formulation of the solution for RF ODE and then applying the high-order Taylor expansion. This has not been explored by previous works. If we apply 3rd order expansion, the performance can be further improved (Table5).

RF-Edit

The methods you mentioned are based on U-Net. Among them, Dragdiffusion and Style Inject target point-based editing and style transferring, which is not the core focus of our work. Due to the time limit, we would like to further explore the potential of RF-Edit on these tasks in future work.

Our work focuses on prompt-based editing using DiT (such as FLUX and OpenSora). Given DiT's distinct architecture from U-Net and larger parameter count, designing an effective and efficient feature sharing method is non-trivial and underexplored. Addressing this, we thoroughly explore the different choices of feature sharing in DiT (more results are shown in the "More Experiments" Section), proposing RF-Edit. RF-Edit is a unified feature-sharing-based framework which can be applied to various DiT architectures. Achieving satisfying results, we believe RF-Edit is insightful for further work.

More Experiments

Thanks for your valuable and insightful advice! More experiments are provided as follows:

• Step-vs-quality analysis: Shown in Table5.
• Alternative metrics for image editing: Shown in Table6. At the same time, we would like to kindly point out that LPIPS is also a widely-used metric for measuring the image consistency in previous works of image editing such as PnP (CVPR 2023), Null-Text-Inversion (CVPR 2023).
• Choices for feature sharing: Further analysis is shown in Figure11, Figure12, Figure13, Figure14. We finally choose to share the V feature in last 20 single-stream blocks of FLUX in the timesteps near to noise, which is proved to be the most effective choice.
• Comparisons on more baselines: Results about MasaCtrl are shown in Table6 and Figure15. Results about SANA are shown in Table8.

Detailed Experiment Setup

For image editing task, our evaluation dataset consists of about 300 images, following previous works such as PnP (CVPR 2023), InstructPix2Pix (CVPR 2023), Null-Text-Inversion (CVPR 2023) and SmartEdit (CVPR 2024). For the image generation task, we conducted experiments on MS-COCO validation dataset, following previous works such as Stable Diffusion (CVPR 2022). For video inversion and editing tasks, we mainly follow the dataset construction process in COVE (NeurIPS 2024).

For image generation task, the NFE is set to 10 for both RF-Solver and Vanilla RF. The timestep for DPM-Solver++ is set to 20 according to their official github repo. For editing tasks, the hyperparameters for all the baseline methods follow their official github repo for best results. We will add more detailed information in the main paper.

Other Questions

Doubling baseline solver ... pareto-optimality in lower number of steps.

As illustrated in Table5, with the increase of timesteps, the performance of both baseline and our methods becomes better. In low timesteps scenarios such as 3 steps for RF-Solver-2 (6 NFE) and 7 steps for Vanilla RF (7 NFE), our methods also illustrate better performance.

Discuss potential failure ... computational cost.

The inversion prompt is optional and does not significantly impact editing results (you can also see from some examples provided in the supplementary materials) and our method can edit a high-resolution image (1360*768) with less than 1 minute on a single A100 GPU. The failure case is shown in Figure16. We will add these discussions to the main paper！