We thank the reviewer for the constructive and detailed comments. We are glad that the reviewer finds our work clear and well-motivated with detailed results. Below, we address the reviewer’s questions one by one.

Comment 1: ...The current paper does not address these problems convincingly for the following two reasons:

-- First, both numerical examples are low-dimensional so that the curse of dimensionality does not affect much.

-- Second, their dynamics are not very complex. ...I was wondering if there is a chance to obtain a good solution even if the simulation of fluid dynamics is not accurate. If this is the case, the results of this example may not provide strong evidence that the algorithm can handle complex dynamics. Additionally, this paper does not present the error analysis for the fluid dynamics simulation, only stats of energy comparison are provided.

Answer: Our tasks align with existing data-driven physical control papers [1-3]. Controlling Burgers' Equation is challenging due to shock wave tendencies, and our setup is more difficult with partial observation/control combinations. The 2D jellyfish control task includes challenges such as (1) asymmetric wake vortex formation from symmetric structures and motion modes [4]; (2) the nonlinear, complex interaction of vortices with the shell [5].

Still, to make our experiments more convincing, we conducted new 2D incompressible fluid control experiments, following a similar (but more challenging) setup of [3]. Control forces were applied within a 64x64 grid flow field, excluding a semi-enclosed region, to minimize the smoke failing to exit through the top middle exit (seven exits in total). For illustration of settings, please refer to subfigures (a) and (b) of Figure 1 in the PDF file at the end of General Response. This high-dimensional indirect control problem involves managing 2-dim forces at approximately 1,700 grid points per time step, resulting in about 100,000 control variables in total across 32 time steps, making it highly challenging. We generated 20,000 training trajectories (including features of the smoke density field, velocity fields, control force fields and smoke proportion field) after filtering low quality trajectories. The average control objective in the training set is 49.9%. We evaluated 50 test samples, and the results are shown in the following Table 1. It shows that our method still has significant advantages, especially in validating the role of prior reweighting. The average relative L2 error of fluid dynamics simulation of our method is 19.2%. In particular, the curse of dimensionality is effectively addressed by our method. For visualization of the generated control and density map of smoke, please refer to subfigure (c) of Figure 1 in the PDF file at the end of General Response. These updates will be added to the final version of our manuscript.

Table 1: Performance comparison on the new 2D indirect fluid control task.

Comment 2: This algorithm depends on the data generated by finite difference methods, hence suffering from the curse of dimensionality;

Comment 3: Is it possible to make the proposed algorithm independent of classical methods (e.g., the finite difference method used in this paper) for data generation?

Answer: Our method does not depend on the data generated by finite difference methods. Our method belongs to the domain of deep learning-based surrogate modeling, and like any other methods in this domain, is decoupled from the data-generating process. The data can come from classical solvers (not limited to finite difference methods) or actual physical observations, e.g., 1D data from finite difference methods while 2D data from the finite volume method [6] in our paper.

Moreover, like any other deep learning-based surrogate modeling method, the goal is exactly to deal with complex, high-dimensional processes. It is exactly our generative control method that can better learn the high-dimensional dependencies from the given data.

Comment 4:What is the energy term $E_\theta (u,w,c)$?

Answer: It is the parameterized energy-based model of the joint distribution of $p(u, w, c)$, where $w$ is the control sequence, $u$ is the state trajectory, and $c$ is the set of conditions, like initial conditionals and boundary conditions. $E_\theta(u,w,c)=-\log(p(u,w,c))+\text{const.}$ characterizes $p(u,w,c)$. Given $c$, we aim to generate $(u,w)$ that lies on the data manifold with high probability, with the optimal objective $J$. To achieve this goal, we train a diffusion model $\epsilon_\theta$ to approximate $\nabla E_\theta(u,w,c)$. During inference, the sampling starts from a Gaussion noise $(u_K, w_K)$ and travels along $\nabla E_\theta(u,w,c)$ by applying $\epsilon_\theta$ iteratively to achieve a final sample $(u_0,w_0)$ with near-minimal energy $E_\theta(u_K, w_K, c)$, under the guidance of $J$.

Comment 5: In the 2D Jellyfish task, what is the value of the kinematic viscosity ν?

Answer: In our paper, ν is set to 0, considering an inviscid fluid, similar to other flow field control papers [3]. Our method can also be applied to viscous fluids and does not restrict the magnitude of ν.

[1]Solving pde-constrained control problems using operator learning,2022.

[2]Optimal control of PDEs using physics-informed neural networks (PINNs), 2021.

[3]Learning to Control PDEs with Differentiable Physics, 2020.

[4]Stable hovering of a jellyfish-like flying machine, 2014.

[5]Propulsive performance and vortex dynamics of jellyfish-like propulsion with burst-and-coast strategy, 2023.

[6]Conservative Volume-of-Fluid method for free-surface simulations on Cartesian-grids, 2010.

We appreciate the reviewer’s valuable feedback and helpful suggestions. We are pleased to hear that the reviewer finds our paper well-written, well-motivated, and extensive in its results. Below, we address the reviewer’s questions one by one.

Comment 1: ... Thus, I suggest to revise the introduction to claim planning as the objective. For more information about motion planning, the authors are referred to Chapter 12 (Motion Planning for PDEs) from the reference: Krstic, M. and Smyshlyaev, A., 2008. Boundary control of PDEs: A course on backstepping designs. Society for Industrial and Applied Mathematics.

Answer: Thanks for this constructive suggestion. Based on your advice and the discussions in the referenced literature, our problem setting indeed aligns more closely with a planning task, where future multi-step actions are generated at once. We will revise the introduction and other relevant sections of the paper to change the keyword "control" to "planning".

Comment 2: In line 72 of page 2, the authors claim "...DiffPhyCon facilitates global optimization of long-term dynamics...". It is not clear how the globality (or even non-triviality) of near-optimal control sequence is guaranteed to be achieved.

Answer: We apologize for not clearly explaining global optimization. By this, we mean treating the state trajectory and control sequences over all physical time steps as a single variable during diffusion/sampling. This approach is chosen because control objectives are typically defined over the entire time span. In particular, when the objective function includes several conflicting terms, our method can relieve the myopic failure modes [1] that may exist in reinforcement learning's iterative decision-making. For instance, in our 2D control task, maximizing the jellyfish's average speed requires sharp angle changes in the early stage, conflicting with minimizing the total energy cost $R(w)$. More details are in Lines 302-307 and Appendix H.7 on myopic failure modes of SAC in our original submission.

Comment 3: I am not sure why Algorithm 1 would not yield local optima

Answer: We do not guarantee that Algorithm 1 yields local optima. However, our further theoretical study during rebuttal shows that prior reweighting (Line 6 of Algorithm 1) enhances the probability of obtaining global near-optimal solutions. Informally speeking, for tasks with suboptimal training control sequences, there exists a hyperparameter $\gamma < 1$ such that using prior reweighting with this $\gamma$ increases the likelihood of obtaining near-optimal control sequences compared to not using this technique ($\gamma = 1$). Due to space limitations, for details of the formal statement of the theorem and proof, please refer to Official Comment on top of this webpage.

[1] Janner, et al. "Planning with diffusion for flexible behavior synthesis.", ICML 2022.

We thank the reviewer for the insightful comments. We are glad that the reviewer recognizes the novelty, clarity, results, and significance of our work. Below, we address the reviewer’s questions one by one.

Comment 1. How is the offline dataset generated? Does it include the optimal control trajectory? What if the offline collected training control trajectory is too far from the optimal one? I am trying to understand if your model "finds" the existing trajectory in the dataset or if it "stitches" different pieces of trajoetiers and generates one that doesn't exist in the dataset.

Anwser: As detailed in Appendices C.1 and D of the original submission, for our 1D and 2D data, we input initial conditions and control sequences with random variations into the solvers to generate trajectories. For instance, in the 2D case, control angles are cosine curves with varying amplitude, mean, and duty cycle. These random variations are necessary as expert control sequence data are hard to obtain for nonlinear and high-dimensional physical systems. Consequently, generated sequences are typically far from optimal.

This issue motivates our prior reweighting technique, which reduces the weight of prior control sequences to increase the probability of sampling near-optimal solutions. In this rebuttal process, our additional theoretical analysis shows that for tasks with suboptimal training set control sequences, using prior reweighting with 𝛾<1 increases the likelihood of obtaining near-optimal control sequences compared to not using it (𝛾=1). Due to space limitations, for details of the formal statement of the theorem and proof, please refer to Official Comment on top of this webpage.

Despite the offline training control sequences not being optimal, diffusion models can combine segments of these trajectories through conditional generation or control target guidance, as illustrated in [1]. In our experiments, a close look at Figure 4 reveals that although the entire curve has not appeared in the training set, its segments resemble parts of a cosine control curve from the training set.

Comment 2: For the cases in this paper, using the online control algorithm is not too expensive. Online RL for these cases could also give global optimal control results. Could you compare the computational cost of your proposed method and an online RL method to demonstrate the benefits against the online RL algorithm?

Anwser: First, online training of SAC on our 2D Jellyfish control case is so computationally demanding that it is hardly feasible. However, we conducted a new experiment using SAC on our 1D Burgers' control task. The results are shown below. Our method is about twice as fast as SAC and performs better.

Second, the cases in our paper are for evaluation purposes. Our method can be applied to more complex tasks, such as the new high-dimensional indirect fluid control task (see Figure 1 in the PDF of General Response), where online RL training is also infeasible here due to high costs. Moreover, online RL is impractical in many settings, like dangerous exploration (e.g., autonomous underwater vehicles) [2], whereas offline learning can avoid these issues. Thus, our method has broader applications compared to online RL.

Comment 3: For the diffusion-generated trajectory, how did you get Figure 4? Did you input the diffusion-generated trajectory to the numerical solver (Lilly-Pad) to get the physics simulation result? Or did you take the diffusion-generated flow field result as the "physics trajectory"?

Anwser: Figure 4 shows the control results of three jellyfish, each with different initial conditions. Control sequences obtained from each method are input into Lilypad solver to simulate the flow field trajectories, based on which we calculated the average movement speed and control objective $J$.

Therefore, we did not use the flow field trajectories generated directly by the diffusion model as the "physical trajectory". The reason for this is that the evaluation of these control methods, both our method and the baselines, should be based on simulated trajectory of the same solver, to compare them in a fair footing. Details are provided in Appendix E.4 (Line 853 to Line 864) of our original submission.

Comment 4: I didn't quite get the intuition behind Figure 2. Why does reducing 𝛾 to less than 1 shift the red point from margin to center?

Anwser: We apologize for any confusion. In the figure, the red dot at the margin indicates a local minimum of $J$, while the red dot at the center represents the global minimum. In the joint distribution $p(u,w)=p(w)p(u|w)$, we assume the global minimum of $J$ has a lower probability than the local minima, as global optimal trajectories rarely appear in the training dataset. By setting 𝛾<1 and then normalizing by a constant $Z$, the distribution $p(w)^\gamma p(u|w)/Z$ flattens $p(u,w)$, increasing the probability of sampling at the global minimum. Thus, the figure illustrates this ideal situation: with 𝛾=1, we sample a local minimum by using DiffPhyCon-lite; with 𝛾<1, we sample the global minimum. However, this ideal scenario is only for intuitive illustration. Rigorous theoretical analysis shows that with 𝛾<1, the probability of sampling near the global minimum increases compared to 𝛾=1. Due to space limitations, for details of the formal theoretical analysis, please refer to Official Comment on top of this webpage.

[1] Janner, et al. Planning with diffusion for flexible behavior synthesis., ICML 2022.

[2] Levine, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv:2005.01643, 2020.

Thanks for providing a detailed rebuttal to my questions and general proof. The quality and relevance of the rebuttal resolved my concerns. Together with the high-quality manuscript, which involves rich detail, it is definitely a paper beyond the acceptance level. Therefore, I would raise my score to 8. I would strongly encourage the author to publish the final version of the code to facilitate future research.

We thank the reviewer for the helpful comments. We are glad that the reviewer appreciates our jellyfish results and dataset contribution. Below, we address the reviewer’s questions one by one.

Comment 1: The experiments are relatively limited - only include one 1D example and one 2D example.

Answer: Our two tasks align with existing data-driven physical control papers [1-3]. For further evaluation, we conducted an additional 2D incompressible fluid control experiment using a setup similar to [3]. Control forces were applied within a 64x64 grid flow field, excluding a semi-enclosed region, to minimize the smoke failing to exit through the top middle exit. This high-dimensional indirect control problem involves managing 2-dim forces at approximately 1,700 grid points per time step, resulting in about 100,000 control variables in total across 32 time steps, making it highly challenging.

We generated 20,000 training trajectories, with average control objective 49.9%. We evaluated 50 test samples. The results (Table 1) show that our method still has significant advantages, especially in validating the role of prior reweighting. For visualization of the results, please refer to Figure 1 in the PDF file in General Response. These updates will be added to the final manuscript.

Table 1: Performance comparison on the new 2D indirect fluid control task.

Comment 2: I have minor reservations about the novelty of the paper since the primary method relies on the existing diffusion model.

Answer: Our method, while relying on diffusion models, introduces the following three key innovations:

1. New Application Area: We leveraged diffusion models' advantages, such as ease of generalization and global optimization over time, for physical system control. This establishes a new application area for diffusion models.
2. Prior Reweighting Technique: We addressed the challenge of generating control sequences that outperform those in the training set by introducing prior reweighting. This is a novel technique in the diffusion model literature, marking a significant technical contribution.
3. Theoretical Analysis: During the rebuttal, we conducted additional theoretical analysis: for tasks where training set control sequences are far from optimal, using a 𝛾<1 in prior reweighting increases the probability of obtaining near-optimal control sequences compared to not using the technique (𝛾=1). This aligns with our intuitive explanation in the original manuscript and will be included in the final manuscript. Due to space limitation, for formal theoretical analysis, please refer to Official Comment.

Comment 3: Compared with other methods, does the proposed method use less, similar, or more time to train (for the learning-based methods) and inference?

Answer: The efficiency comparison for inference on two tasks was detailed in Appendix I of the original submission. Those resutls are combined with training time comparisons in the following Table 2 and Table 3. Here are the key points:

• Inference Time: Our method is competitive among most methods, except SAC. By adopting the fast sampling method DDIM (DiffPhyCon-DDIM), our method's efficiency significantly improves, nearing the fastest models in the 1D task.
• Training Time: The training time for our method is smaller (1D task) or comparable (2D task) to other learning-based methods.

These details will be included in the final manuscript.

Table 2: Training and inference time comparison on 1D task.

Table 3: Training and inference time comparison on 2D task.

Comment 4. Is it easy or hard to extend this method to 3D systems? In the 2D example, a 3D U-Net is used. Does this make it hard to apply this method to 3D systems as it requires a much larger 4D network?

Answer: Our method is agnostic to the neural network backbone of the diffusion model. For a 3D system, we can switch to an appropriate neural network like a 4D U-net.

Regarding efficiency, controlling 3D complex physical systems is challenging for all current methods. Although autoregressive models require only a 3D neural network, they evaluate iteratively at each physical time step [1,4]. Our method improves temporal efficiency by generating the full trajectory simultaneously.

Comment 5. In the prior reweighting method, are 𝜖𝜃 and 𝜖𝜙 trained jointly, or is 𝜖𝜃 trained first and then 𝜖𝜙?

Answer: These two models can be trained simultaneously as they are independent.

[1]Solving pde-constrained control problems using operator learning, 2022.

[2]Optimal control of PDEs using physics-informed neural networks (PINNs), 2021.

[3]Learning to Control PDEs with Differentiable Physics, 2020.

[4]Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control, 2018.