We thank your recognition of our work's extensive evaluations and your constructive suggestions to improve the presentation of our work. We apologize for some not-well-explained notations with our assumption that all reviewers were familiar with generative models, especially flow matching. We have included a more detailed mathematical introduction to flow matching in the common rebuttal session, and we will further elaborate on some additional details to address your questions and concerns.

Q1 Notations

We first clarify some notations in addition to Section 3.1. We use $u:\mathcal{X}\subseteq\mathbb{R}^d\to \mathbb{R}$ to denote a continuous function as the solution. The subscript always indicates the time step in the flow. We follow the flow matching convention such that $t=0$ corresponds to the noise and $t=1$ corresponds to the target (data). As the space of continuous functions has an infinite dimensionality, we follow FFM [1] to adopt measure-theoretical terms of $\mu_0,\mu_1$ as probability measures over the continuous functions, instead of using the common $p_0,p_1$ in computer vision domains. All $p,q$ in our work are probability densities defined as the Radon–Nikodym derivative (see the FFM paper for technical details). Specifically, at each time step $t$, the noised data $u_t:=\psi_t(u_0|u_1)$​​ can be obtained with interpolation. We noted that all of the above definitions are standard notations in flow matching context.

We do agree Equation 2 is less well-defined as we postponed the definitions in the bulletin list later. We will modify the order in our revised manuscript to first introduce and define the notations to avoid potential confusion. We will also follow your suggestion to move Proposition 1 to the main text to make it more understandable.

Q2 Algorithm

Your understanding of our Algorithm 2, the core algorithm for ECI sampling, is definitely correct. We originally put it in a more concise way mainly because all notations are quite standard in diffusion or flow-based generative models. For example, [2, 3] also adopts an iterative formulation for guiding diffusion models with a double loop. We will consider reformatting the algorithm to include more details following your outline.

Still, we would like to mention that both extrapolation and interpolation stages for flow matching have clear definitions with respect to the probability path induced by the flow. Specifically, the linear interpolation corresponds to the OT-path (OT for optimal transport) in flow matching literature [4] with theoretical benefits of optimal transport. It is not our whimsical assumption but theoretically a good option in choosing the probability paths.

Also, due to the iterative nature of flow sampling, ECI sampling needs to advance the solver for the flow ODE (see common rebuttal session). Therefore, the last step of advancing the solver to a large $t'=t+1/N$​ is also necessary.

Q3 Other Remarks

• Regarding $\mathcal{F}_\phi$. Your understanding is correct. We will clarify it.
• Regarding the domain of $x$. We assume the normal domain of $\mathcal{X}$ for $\mathcal{F}_\phi$.
• We used the notation $\mathcal{U_{|F}}$ to indicate the solution set is restricted by the constrained to a narrower subset.
• We used $:=$ for definition (of new quantities). For the push-forward, we use the standard notation of $(\psi_t)_*$​, e.g., the second part of Section 3.1.
• For the correction algorithms in Algorithm 3&4, they serve as concrete examples of the correction $C(u_1,\mathcal{G})$ in Equation 3&4 and Algorithm 2.

We also thank your other suggestions regarding the notations and formatting for better conciseness and understandability. We will add more clarifications on notations, more mathematical backgrounds, and better presentations in our revised manuscript to make our work more accessible to a larger audience.

[1] Kerrigan, Gavin, Giosue Migliorini, and Padhraic Smyth. "Functional flow matching." arXiv preprint arXiv:2305.17209 (2023).

[2] Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Kawar, Bahjat, et al. "Denoising diffusion restoration models." Advances in Neural Information Processing Systems 35 (2022): 23593-23606.

[4] Lipman, Yaron, et al. "Flow matching for generative modeling." arXiv preprint arXiv:2210.02747 (2022).

We thank your recognition of our work's comprehensive experimental results and concise formulation. We are happy to address your questions and concerns. Still, we noted the potential misinterpretation of our proposed approach as a regression model but not as a generative model. See the common rebuttal for more details, clarifications, and mathematical backgrounds.

Q1 Results in Figure 6-8

Figure 6-8 demonstrates the generation statistics of point-wise mean and standard deviation, together with the ground truth statistics. Similar to Figure 1&2, a closer resemblance to the ground truth statistics indicates a better generation result. It is worth noting, though, that MMSE and SMSE only reflect the first two moments of the distribution but not higher-order information. Therefore, FPD scores should be preferred, in which the pre-trained PDE foundation model Poseidon is used to extract the hidden representations (see Appendix D.1). We have included the detailed experimental setups in Appendix B and we will add more explanations for Figure 6-8.

Q2 Experiments with different IC/BC

Thanks for the insightful suggestion. We are running experiments on the Stokes dataset with different IC/BC values to further demonstrate the generalizability and effectiveness of our approach. We will update the results once they are readily available.

Q3 Overall framework

The overall framework of our ECI sampling can be understood as a modification to the vanilla flow sampling process in Algorithm 1. With minimum modifications, we demonstrate the ECI sampling in Algorithm 2. Previous work in guiding diffusion models also adopts an iterative formulation [1, 2], which is in fact standard in the context of diffusion or flow-based generation models. We understand that the reviewer may not be familiar with the generative models and we have provided more details in the common rebuttal session.

Regarding the datasets used in our work, we noted all datasets and their spatiotemporal resolution setups are directly taken from previous papers (see Appendix B for reference of the datasets in previous papers). Therefore, we believe such an experimental setup is standard, consistent, and comparable with existing works. To the best of our knowledge, 4D datasets in PDE systems do not exist so far. It is also a standard practice to test on 2D and 3D datasets in the PDE domain, e.g., neural operator learning [3, 4], foundation models [5, 6], and existing constrained generation [7, 8]. Therefore, we believe we have already provided enough evidence of the superior performance of our proposed method.

We already included the dataset descriptions, specifications, and task formulation in Table 2&7, Appendix B&D.2, for which our ECI sampling serves as a unified constrained generation framework.

Q4 Improvement in the whole domain

We note that our ECI sampling was designed to offer iterative control in each flow sampling step to mitigate the artifacts between the constrained and unconstrained regions. Indeed, our generation result demonstrated remarkable consistency in the whole spatiotemporal domain, with hardly noticeable artifacts around the boundary in Figure 1, 6, 7, and 8. This is in sharp contrast with the naive projection algorithm like ProbConserv, which can be only applied to the final prediction without fine-grained control over each sampling step. In the close-up demonstration in Figure 5, noticeable artifacts can be found for ProbConserv and other gradient-based approaches.

The iterative control of ECI sampling that gradually pushes the intermediate noised data into generations with hard constraints can be further demonstrated in Figure 4. Note how the generation trajectory gradually exhibits more consistency between the constrained and unconstrained regions.

Q5 Zero-shot effectiveness

We emphasize that all experiments are carried out in a zero-shot manner except for the conditional FFM baseline. Specifically, an unconstrained FFM was first trained as the generative prior. Different approaches including our ECI sampling were then applied in a zero-shot manner (see Appendix C&D.2). In this way, our framework provides a unified approach for different constraints without finetuning or conditional training (e.g., IC and BC in the Stokes problem). The generative evaluation metrics also demonstrated the state-of-the-art performance across all zero-shot models (conditional FFM is not zero-shot and should be viewed as an upper bound, as we have discussed in Section 4.1).

Thank you for your appreciation of our work's clear presentation, sufficient ablation studies, and effectiveness regarding hard constraints. We will address your concerns as follows.

Significant misunderstanding and misinterpretation of our work

We would first like to point out a significant misunderstanding and misinterpretation of our work. We address the constrained generation of the flow matching framework. The guided flow matching model during the iterative sampling process is still a generative model whose output is a distribution over solutions instead of a single ground truth solution. Sadly, we noted the reviewer emphasized the projected gradient approaches which are infeasible for flow-based generative models. The reviewer also misinterpreted our generative evaluation metrics and emphasized constrained models in regression models like projected gradient, which was not our claimed contribution.

We further distinguish our problem and experimental setup from existing work, especially for the projected gradient-related methods, in the following aspects.

• Generative modeling. Unlike projected gradient-related methods, our proposed framework is a generative framework that captures the distribution of solutions instead of a single solution.
• Challenge. The challenge for guiding flow (or diffusion) based models lies in its iterative sampling process, as we have mentioned in the second part of related work and Table 1. The iterative nature makes it infeasible to employ normal projected gradient approaches.
• Gradient-free guidance. Our proposed ECI-sampling is completely gradient-free. We demonstrated in our experimental results that our gradient-free approach still outperformed gradient-based methods while achieving significant speed-up.
• Evaluation metrics. The generative nature requires different evaluation metrics that capture distributional properties instead of simple MSE to the ground truth. In this regard, we calculate the point-wise MSE of the mean and standard deviation between the ground truth distribution and the generation distribution. We further apply Poseidon, a PDE foundation model to evaluate the Frechet distance.

We further reiterate our contributions, which we believe may have also been misinterpreted by the reviewer, as follows:

• We proposed ECI sampling for hard-constrained generation for pre-trained flow-based generative models. We emphasize flow-based generative models, which impose unique challenges due to their iterative sampling nature. Existing approaches for constrained prediction in regression models are infeasible or perform badly. To the best of our knowledge, we are the first to deal with the zero-shot flow-based constrained generation problems for functional data.
• The extrapolation and interpolation concepts are unique to flow-based models, which are completely different from any non-iterative approaches like project gradient. Indeed, we do not claim contributions in the correction — we even manually placed it in the Appendix as it is less important. Instead, the iterative control framework unique to flow-based models using the interleaved ECI steps is our major claim for a better generation result.
• Our approach guarantees the exact satisfaction of constraints without the expensive gradient information.
• Our approach achieves state-of-the-art generative performance on most evaluation metrics. We emphasize generative evaluation metrics, as the reviewer has drastically misinterpreted our metrics as regression MSEs. Instead, distributional properties are calculated and the foundation model Poseidon is used to give the most comprehensive metric.

We understand the reviewer might not be familiar with generative modeling in the PDE domain, which, as we have mentioned in the first part of our related work, is a newly emerged realm for functional data. However, as we have already pointed out the reviewer's misunderstanding and have distinguished our problem setup and contributions from existing work, we urge the reviewer to give a more fair and comprehensive evaluation of our work.

I would like to thank the authors for their detailed rebuttal. While I still believe that the theoretical component of the paper is somewhat lacking, the method does present some practical benefit both in terms of hard constraint satisfaction and some performance improvements (as described in the follow-up comments/results).

Overall, I think the paper is well written and presents some empirical evidence, but would benefit from more theoretical analysis and motivation. I have revised my score.

We sincerely appreciate your recognition of our work's high readability and comprehensive experiments and ablation studies. We also thank your recognition of our ECI sampling as a generative model. We will clarify and address your questions as follows.

Q1 Problem Setup

We will further elaborate on our problem setup in more rigorous measure-theoretic terms as follows. Consider a well-posed PDE system $\mathcal{F}\_\phi u(x)=0, x\in\mathcal{X}\subseteq \mathbb{R}^D$ with PDE parameters $\phi\in\Phi$, then the solution map $\mathcal{T}:\phi\mapsto u(x)$ is a well-defined mapping. For a fixed distribution $p_\Phi$ over the parameter $\phi$, we can naturally define the measure over the solution set as the push-forward of the solution map: $\mu\_\mathcal{F}=\mathcal{T}\_* p\_\Phi$. Now consider some constraint operator $\mathcal{G}u(x)=0,x\in\mathcal{X_G}\subseteq \mathcal{X}$ defined on a subset of the PDE domain. Denote the solution set for $\mathcal{G}$ as $\mathcal{U_{G}}$ and the characteristic function over the solution set as $\chi_\mathcal{G}$. With the non-degenerating assumption that $\int_\mathcal{U} \chi_\mathcal{G}d\mu_\mathcal{F}>0$ (this is essentially saying that we should have at least one solution), the conditional measure can be obtained as $\mu_\mathcal{G}(A):=\mu_\mathcal{F}(A|\mathcal{G})=\int_A\chi_\mathcal{G}d\mu_\mathcal{F}/\int_\mathcal{U} \chi_\mathcal{G}d\mu_\mathcal{F}$. We assume $\mu_\mathcal{F}$ can be approximated well by generative priors like FFM, and we want to guide it towards the conditional measure $\mu_\mathcal{G}$.

It is straightforward from the above measure-theoretic formulation that whether the measure $\mu_\mathcal{G}$ over the constrained solution set $\mathcal{U_{G}}$ is uniform depends on the prior choice of the distributions over PDE parameters $p_\Phi$. In our current experiments, all PDE parameters are sampled uniformly over a fixed interval (see Table 7 and Appendix B for details). This setting indicates the target measure over $\mathcal{U_{G}}$ is indeed uniform in our case. However, we can easily alter the parameter distribution, e.g., sampling some PDE parameters from the standard normal distribution instead. In this way, the target measure over the solution set is non-uniform but transforms accordingly with the push-forward.

The problem setup can be understood more intuitively, as shown in the example of the Stokes problem demonstrated in Figure 5 in our updated manuscript. In the unconstrained pre-trained stage, the FFM is trained with both variable BC and IC. During zero-shot constrained generation, either IC (middle) or BC is fixed (right) but not both of them. In this way, solutions in the middle figure share the same left-column values while the variable BC still allows for a family of solutions. Similarly, solutions in the right figure share the same top-row values with variable IC as a family of solutions.

Q2 ECI's Advantages and Comparisons

We have briefly discussed the potential advantage of ECI sampling in Section 3.2 and we further provide a more systematic summary here.

• Compared to existing correction algorithms for regression NOs (e.g., ProbConserv), ECI sampling offers fine-grained iterative control in the flow sampling process, leading to a more consistent generation. As demonstrated in our visualizations, directly applying the correction algorithm to the final generation will lead to noticeable artifacts. ECI sampling, on the other hand, can generate predictions with hardly noticeable artifacts.
• Compared to existing gradient-based algorithms (e.g., DiffusionPDE, D-Flow), ECI sampling adopts a gradient-free approach that ensures the exact satisfaction of the constraints while enjoying a significant speedup and memory saving (see our common rebuttal). We also mentioned in Section 1 why gradient-based methods may fail in PDE scenarios.
• Similar to some previous work (D-Flow, RePaint, DDRM), ECI sampling can be also applied recursively in a single sampling step, leading to better information mixing between the constrained and unconstrained regions. This is because the correction information is "propagated" back to the current timestep via interpolation, and multiple rounds of correction can potentially improve consistency with the constraints.

These points are further supported by our extensive experiments over various 2D and 3D PDE systems, in which our ECI sampling was able to achieve state-of-the-art performance on most generative metrics.

We already provide a fairly comprehensive related work in Section 2, where existing zero-shot guided generation models in computer vision domains are also included. An itemized comparison with the existing baselines is provided in Table 1, and further details regarding the baselines are provided in Appendix B.

We appreciate your recognition of our work's clear motivation in scientific domains, the benefit of our zero-shot formulation, and competitive performance. We will address your questions and concerns as follows.

Q1 ECI Algorithm

We thank your suggestion. As we have mentioned in our general rebuttal, we have updated the revised manuscript with clearer definitions of notations, an additional description of the ECI step in Algorithm 2, and more explanations to better describe our algorithm in a more concise and accurate way.

Q2 Contribution

We formulate our contributions more clearly in the following sense:

• We proposed ECI sampling for hard-constrained generation for pre-trained flow-based generative models. We emphasize flow-based generative models, which impose unique challenges due to their iterative sampling nature. Existing approaches for constrained prediction in regression models are infeasible or perform badly. To the best of our knowledge, we are the first to deal with the zero-shot flow-based constrained generation problems for functional data.
• Our proposed ECI sampling is an iterative control framework unique to flow-based models using the interleaved ECI steps. Such a fine-grained iterative control better facilitates information flow between the constrained and the unconstrained regions, leading to a more consistent generation with fewer artifacts.
• Our approach guarantees the exact satisfaction of constraints without the expensive gradient information. Such a gradient-free formulation achieves significant speedup and memory-saving compared to existing gradient-based approaches. In this way, our framework enjoys easy scalability with little time or memory overhead.
• Our approach achieves state-of-the-art generative performance on most evaluation metrics for zero-shot constrained generation. We emphasize generative evaluation metrics, for which distributional properties are calculated and the foundation model Poseidon is used to give the most comprehensive metric of FPD.

The changes are also reflected in our revised manuscript.

Q3 Supply Chain Applications

We noted that our framework, in principle, can be applied to different functional data including those in supply chain applications. However, we also noted similar data are scarce in such a domain, so we instead instantiate our approach on PDE domains where more well-established data are available and baseline-constrained generation models are also available. We will leave additional applications in other scientific domains as future work and we have carefully modified our manuscript to avoid overclaiming our contributions.

Q4 Vanilla FFM Performance

As a sanity check, we provide the generative metrics of the pre-trained FFM without any guidance in the table below. It can be demonstrated that there is indeed a shift in the distribution from the generative prior learned by the unconstrained FFM, as many errors are large. Our proposed ECI sampling can successfully capture such a shift in the distribution with significantly lower MSEs and a closer resemblance to the ground truth constrained distribution.

Q5 Resampling & Stochasticity

We further elaborate on the effect of the resampling interval length and its relationship to sampling stochasticity. We first noted that the "magnitude" of the distribution shift is different for different constraints. This can be also understood by the results in Q4, where tasks like Stokes IC have smaller original FPDs, but tasks like Stokes BC have larger ones. We noted that such a property is intrinsic to the constraint. Therefore, it might be difficult for a single framework to deal with different magnitudes of distribution shifts simultaneously. In this sense, stochasticity during our constrained sampling should be larger for tasks with larger distribution shifts to guide the model toward further exploration, and vice versa.

With this observation, we introduced the length of the resampling interval as a tunable hyperparameter to offer control over such stochasticity for each individual task. In this way, when the stochasticity matches those in the task, the model is easier to be guided toward the shifted distribution with more consistency and thus better generation results. We also provided a discussion of some heuristic rules for choosing this value in Section 3.3. However, a more theoretical analysis is always preferred, as we also mentioned as our limitation in Section 5.

Scalability

We first noted that all datasets and their spatiotemporal resolution setups are directly taken from previous papers (see Appendix B for reference of the datasets). Therefore, we believe such an experimental setup is standard, consistent, and comparable with existing works. It is also a standard practice to test on 2D and 3D datasets in the PDE domain, e.g., neural operator learning [1, 2], foundation models [3, 4], and existing constrained generation [5, 6]. Therefore, we believe we have already provided enough evidence of the superior performance of our proposed method. Furthermore, as our proposed ECI sampling is model-agnostic, the time and memory bottleneck lie with the vector field encoder, which is parameterized by some neural operator. In this work, we used the Fourier neural operator, but any other new encoder architecture can be readily integrated within our framework to further boost the scalability of the overall framework.

We further provide extensive benchmarking of the sampling time and GPU memory usage of different sampling methods on the 2D (Stokes IC) and 3D (NS equation) datasets in the table below. Unfavorable values are highlighted in bold. Compared with gradient-based methods, our gradient-free approach enjoys sampling efficiency in both 2D and 3D cases with little overhead compared to the original flow sampling. Noticeably, ECI-1 was able to achieve x440 acceleration and x310 memory saving compared to gradient-based D-Flow on the Stokes problem. Such a significant advantage of computation and memory efficiency ensures the scalability of our proposed ECI sampling to larger datasets or data points where the gradient-based approach may fail.

To further provide a more concrete example of scaling up to higher resolutions, we adopt the zero-shot superresolution setting to test all of the methods with a generation resolution of 200×200 on the Stokes problem. The results are summarized in the following table:

D-Flow runs out of memory even with a batch size of 1. On the other hand, our ECI sampling continues to achieve the best generation results, though slightly worse than its 100×100 resolution counterpart. It is also interesting to see that the conditional FFM model is the most sensitive approach, probably because it has hard-coded the resolution information for a better generation performance during training. In this way, we can safely conclude that our framework demonstrates better scalability on higher-resolution data than gradient-based baselines.

[1] Li, Zongyi, et al. "Fourier neural operator for parametric partial differential equations." arXiv preprint arXiv:2010.08895 (2020).

[2] Li, Zongyi, et al. "Anima Anandkumar, Physics-informed neural operator for learning partial differential equations." arXiv preprint arXiv:2111.03794 (2021).

[3] Herde, Maximilian, et al. "Poseidon: Efficient Foundation Models for PDEs." arXiv preprint arXiv:2405.19101 (2024).

[4] Sun, Jingmin, et al. "Towards a Foundation Model for Partial Differential Equation: Multi-Operator Learning and Extrapolation." arXiv preprint arXiv:2404.12355 (2024).

[5] Huang, Jiahe, et al. "DiffusionPDE: Generative PDE-solving under partial observation." arXiv preprint arXiv:2406.17763 (2024).

[6] Négiar, Geoffrey, Michael W. Mahoney, and Aditi S. Krishnapriyan. "Learning differentiable solvers for systems with hard constraints." arXiv preprint arXiv:2207.08675 (2022).