Towards Hierarchical Rectified Flow

Thanks a lot for your time and feedback and for assessing the paper as presenting interesting theorems.

QC1: Can authors provide a detailed table comparing HRF and RF models, including parameter counts, training time per iteration, and memory usage across all experiments?

For synthetic data, we included details regarding the training/inference time in the newly added Appendix G (see Table 2 and Table 3). For MNIST and CIFAR-10, we compared training/inference time and generation performance of the baseline RF and of our method in the newly added Appendix H.3 of the revised appendix (see Tables 4-6). We briefly summarize the results here. For MNIST, our model outperforms the baseline while maintaining a comparable model size, training time, and inference time. For CIFAR-10, although our model is 1.25x larger and has a roughly 1.4x slower inference time, it still achieves superior performance compared to the baseline: 3.713 FID vs. 3.927 FID for the baseline. We think this trade-off is justified.

QC2: Is the reported NFE=J⋅L? What is the time required per NFE in both HRF and RF?

Yes, for HRF with depth 2, NFE=J⋅L. Note, all NFEs in our paper are Total NFEs: $\text{NFE}=\prod_d N^{(d)}$, where $N^{(d)}$ refers to the integration steps at depth $d$. To clarify, we added L349 in the revised paper. Further, all figure axis labels have been updated to “Total NFEs”. To clarify further, we added an ablation study in Appendix G, analyzing NFEs and the impact of integration steps across depths. For synthetic data, training and inference times are reported in newly added Table 2 and newly added Table 3. For MNIST and CIFAR-10, training and inference times are reported in newly added Table 4 and newly added Table 5. Summarizing the tables briefly, training and inference times are on par.

QC3: The results for CIFAR-10 unfortunately are not encouraging. Considering the results on MNIST and CIFAR-10 it seems that the proposed model does not scale as well.

We kindly disagree, on both MNIST and CIFAR-10, our approach achieves better performance using the same total NFEs. For MNIST, our model size is comparable (slightly smaller) while the results are better (see Fig. 6). For CIFAR-10, our model size is 1.25x larger but we achieve a slightly better FID of 3.713 vs. 3.927 (see Table 6 in the revised appendix). We include more details in Appendix H.3 (see Tables 4-6) and updated Figure 6 in the main paper to provide the parameter count.

QC4: ImageNet results

Unfortunately we don’t have the computational resources to conduct experiments on higher-dimensional data like ImageNet. However, our core idea---the hierarchical structure---is theoretically applicable to any (latent) diffusion architecture that utilizes flow matching for learning a vector field. We are open to exploring such experiments in future work and are very excited to collaborate with any interested parties.

QC5: The resulting model is not a diffeomorphism and as such we lose the ability to perform density estimation.

We added Appendix D in the revised version of the paper to show how density estimation can be performed for an HRF model. We observe our model to lead to more accurate density estimation results than the RF baseline (see newly added Figure 7).

QC6: The paper claims that the resulting paths are straight as the trajectories can intersect. However the trajectories are stochastic, and thus they are likely to increase overall length. I would appreciate a clarification from the authors regarding this matter.

In the paragraph starting at L291, we mentioned that our generation process for the data can be piecewise straight and that a large numerical integration step size $\Delta t$ is acceptable. For inference, piecewise straightness matters more than the overall length, because this implies that we can use less numerical integration steps while maintaining accuracy. This reduces the number of NFEs and improves the efficiency. In our experiments, we typically only use 2-5 data integration steps. Results reported in the newly added Tables 3, 5, 6 corroborate this.

I thank the authors for the response. My concerns regarding computational demands have been considerably reduced. However some issues still linger.

QC1:

Time results in Table 2 are not that informative. The authors should use scientific notation to present results as the ratio between time is what matters. Or they can present the overall training time per number of iterations (fractional form), as that is the quantity I requested.

I am not sure why the the same network size was not used in CIFAR experiments like in the MNIST case. That would have enabled a more direct comparison.

QC2:

This was a key concern. The fact that the total number of of steps in both methods is the same indicates that the method is not as computationally demanding as I originally feared.

QC3 & 4:

Unfortunately, we indeed kindly disagree. On MNIST the proposed method significantly outperforms the baseline using the same network size. For CIFAR a larger network is used but the difference with the baseline shrinks. As such it is important to show that the method scales. The type of scaling I am worried most relates to the increase of distribution complexity and not dimensionality. As the distribution of the data becomes more complex (has more modes and greater variability), the distribution of the velocities, as authors show, also becomes more complex. As such, at each integration step, there could be contradictory directions as a result of sampling from a highly varying distribution. Therefore it is important to test on Imagenet, even at a reduced scale of 32x32. This relates to QC6.

QC5:

I mostly meant a path-wise density estimation method. That is, a method that is computationally cheap and that does not rely on Monte-Carlo estimations. As such I prefer Algorithm 3 considerably more than 4. as it does not rely on Equation 22. Regarding Algorithm 3, is there a reason to favor also $\pi_1(0; z_1, 0)$ instead of $\pi_1(z_1-z_0; z_0, 0)$ in Step 3? Then one can calculate

$\log \pi_1(0; x, 0) = \log \pi_0(u_0; x, 0) - \int_1^0 \nabla_{u_\tau} \cdot a_\theta(x, 0, u_\tau, \tau) d\tau$

using the instantaneous change of variable formula of [1] (this reference should be added). So my question is, what role does the sampling of $z_0$ play in the variability? Also reference [2] should be added when using the Hutchinson trace estimator.

From an experimental perspective, I was hoping to see this method applied to 2D complex toy datasets and, more importantly, to observe bits-per-dimension results on image datasets such as MNIST and CIFAR-10. Conducting such experiments would not be particularly computationally demanding and could likely be completed within a few hours on a consumer-grade GPU.

QC6: Since all generated paths are discrete approximations, they are inherently piecewise. As I clarified in QC3 & 4, my concern is that for more complex distributions, the model may struggle to decide on a direction, leading to a pronounced zig-zag pattern resembling Brownian motion, significantly more than in the baseline case. For instance, in Figure 1c, some particles appear to move from the bottom-right toward the upper-left, only to reverse course and head toward the upper-right. My concern is that this behavior could become more pronounced as the dimensionality increases and, more critically, as the distribution's complexity grows, potentially amplifying directional uncertainty.

Notation: I noticed that Theorem 2 is proven for 1D data, as such $k$ should be a vector, and $kx$ should be written as $k^Tx$.

Overall, I believe the appropriate evaluation for this paper is a score of 7. If the main concerns regarding scaling on ImageNet 32x32 and density estimation for at least MNIST are addressed effectively, I would consider the paper deserving of an 8. However, since selecting a score of 7 is not an option, I will assign an 8, recognizing that the paper introduces a valuable new branch for research.

References

Neural ODEs, Chen et al 2018

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models. Grathwohl et al. 2018.

Thank you for the response!

QC5:

Thank you for the additional experiments. I believe attention should be paid to MNIST results. Bits per dim scores are above 2, which is significantly higher than that of FFJORD 2018 (bits per dim of 0.99). CIFAR results are quite typical however. Also, as an advice for the final version of the paper, in the case of 2D data, authors could add 2D density plots like in Figure 2 of FFJORD 2018.

QC6:

Considering that at time $t=0$ the distribution of velocities coincides with a shifted version of the data distribution, there is a possibility that sampling stochasticity will be high when the data distribution is complex. However, this remains to be tested more thoroughly in the future.

Thanks gain for the response, I look forward to seeing the Imagenet results.

Thanks a lot for your time and feedback and for assessing the paper as having a clear and impactful motivation, a solid theoretical foundation, and intuitive experimental demonstrations.

QB1: Ambiguity in Extended Hierarchical Approach

The paragraph below Eq. (10) explains the definition of each vector. In the newly added Appendix E we present a more detailed derivation in order to clarify further.

QB2: Scalability and Computational Complexity

For synthetic data, we included details regarding the training/inference time in the newly added Appendix G (see Table 2 and Table 3). For MNIST and CIFAR-10, we compared training/inference time and generation performance of the baseline RF and of our method in the newly added Appendix H.3 of the revised appendix (see Tables 4-6). We briefly summarize the results here. For MNIST, our model outperforms the baseline while maintaining a comparable model size, training time, and inference time. For CIFAR-10, although our model is 1.25x larger and has a roughly 1.4x slower inference time, it still achieves superior performance compared to the baseline: 3.713 FID vs. 3.927 FID for the baseline. We think this trade-off is justified.

QB3: Scope of Experiments

We added additional experiments in Appendix H.2 and Appendix H.3, demonstrating compelling results on synthetic data, MNIST, and CIFAR-10. Unfortunately we don’t have the computational resources to conduct experiments on higher-dimensional data like ImageNet. However, our core idea---the hierarchical structure---is theoretically applicable to any (latent) diffusion architecture that utilizes flow matching for learning a vector field. We are open to exploring such experiments in future work and are very excited to collaborate with any interested parties.

Thanks a lot for your time and feedback and for assessing the paper as being well written.

QA1: The generation framework demands significantly more model calls compared to the conventional RF framework, specifically NFEs multiplied by the number of discretizations in the velocity space

We think there is a misunderstanding as our model does not require significantly more neural function evaluations (NFEs). Note, all NFEs in our paper are total NFEs, i.e., $\text{NFE}=\prod_d N^{(d)}$, where $N^{(d)}$ refers to the integration steps at depth $d$. For instance, in Figure 4(b), the “HRF2 20 v steps” line at NFE = 200 uses 10 x-steps and 20 $v$-steps ($10 \times 20 = 200$). In contrast, the baseline RF line at NFE = 200 uses 200 $x$-steps. This ensures that the comparison is fair and that our framework does not require more NFEs. To avoid this misunderstanding, we added L349 in the revised paper. Further, all figure axis labels have been updated to “Total NFEs”. To clarify further, we added an ablation study in Appendix G, analyzing NFEs and the impact of integration steps across depths.

QA2: Comparison with rectified RF.

Approaches for straightening the paths in flow matching models are orthogonal to our work. In the original paper, we reviewed those methods at the end of Section 5 and mentioned that they can be adopted in our formulation. To demonstrate this, we have added Appendix H.2 which incorporates OTCFM in our framework. We find that HRF further improves OTCFM. We use OTCFM here since it’s a single-step training approach to straighten the paths.

QA3: Training/Inference time comparison

For synthetic data, we included details regarding the training/inference time in the newly added Appendix G (see Table 2 and Table 3). For MNIST and CIFAR-10, we compared training/inference time and generation performance of the baseline RF and of our method in the newly added Appendix H.3 of the revised appendix (see Tables 4-6). We briefly summarize the results here. For MNIST, our model outperforms the baseline while maintaining a comparable model size, training time, and inference time. For CIFAR-10, although our model is 1.25x larger and has a roughly 1.4x slower inference time, it still achieves superior performance compared to the baseline: 3.713 FID vs. 3.927 FID for the baseline. We think this trade-off is justified.

QA4: ImageNet results

Unfortunately we don’t have the computational resources to conduct experiments on higher-dimensional data like ImageNet. However, our core idea---the hierarchical structure---is theoretically applicable to any (latent) diffusion architecture that utilizes flow matching for learning a vector field. We are open to exploring such experiments in future work and are very excited to collaborate with any interested parties.

QA5: Could the authors provide an ablation study on how model performance and computational requirements change as depth increases?

We included ablation studies in the revised Appendix G. The computational requirements and performance of HRF are mostly independent of the depth, but rather depend on the number of chosen integration/sampling steps. In general, higher depth HRF gives better performance with an acceptable increase in model size, training time, and inference time. For synthetic data please check the newly added Table 2 and Table 3 for details. For MNIST and CIFAR-10, please see the newly added Tables 4-6.

QA6: Network architecture and reproducibility

The network architectures are described in Appendix F.1 and F.2. Note, we revised Appendix F.2 to improve clarity and added Table 2 and Table 4 to compare architectures. We will also release the code with complete hyperparameter settings to ensure full reproducibility of our synthetic examples and experiments on MNIST and CIFAR-10, as stated in L023.

QA7: Training Convergence

We added Figure 8 in Appendix G to show that training of models with different depths remains stable and convergence behavior doesn’t differ significantly from the baseline.

QA8: Minor typo on line 156: it should be \pi_0. Please review and correct typos.

There is no typo in line 156. $\pi_1$ is the velocity distribution at $\tau = 1$, while $t$ is the time variable for the data domain. To clarify we have changed the notation for the velocity distribution $\pi_1(x_t, t)$ to $\pi_1(v; x_t, t)$ throughout the paper. We have also checked the paper again and corrected typos.

Thank you for your response. I appreciate the authors' efforts in revising the paper and addressing the concerns raised. I find the theoretical insights to be interesting and believe they hold potential for meaningful contributions to the field. While the paper may have certain practical limitations, particularly when applied to large datasets with greater depths, the performance of the 2-depth version is comparably strong without requiring excessive resources. Based on this, I will raise my score to 6.