CL-DiffPhyCon: Closed-loop Diffusion Control of Complex Physical Systems

Comment 2: The contributions are unclear. A careful rewriting of the manuscript seems necessary to clarify contributions, and to appreciate the differences between the propose method and the existing ones.

Answer: (Continued from previous piece of response)

Differences from Existing Methods:

Existing methods are discussed in Section 2 of our original submission and can be categorized as diffusion-based control, reinforcement learning, imitation learning, and classical control methods such as PID and MPC. Below, we further clarify the distinctions:

1. Diffusion-based Control Methods (DiffPhyCon, Adaptive Replanning):

   • Unlike existing diffusion-based control methods like DiffPhyCon and adaptive replanning, our approach incorporates real-time feedback, achieving a closed-loop structure rather than an open-loop or purely replanning-based approach. If DiffPhyCon were to conduct replanning at every physical time step, it could achieve closed-loop control; however, the sampling cost would be approximately $H$ times higher ($H$ is the model horizon, i.e., the window size of physical time steps) than our method, as compared intuitively in the left and right subfigures of Figure 1.
   • Unlike these methods, which typically require full synchronous denoising inside the diffusion model horizon at each physical time step, we use an asynchronous denoising framework that significantly reduces computational costs and ensures timely response, making our approach practical for real-time applications.
   • A further comparison with adaptive replanning (RDM) demonstrates that our method is inherently closed-loop (RDM is open-loop), more efficient (as RDM involves extra likelihood computations and more sampling steps), and easier to use (RDM requires tuning two hyperparameters extensively for new tasks). These differences have been discussed in Section 2 of our original submission.

2. Reinforcement Learning (RL) and Imitation Learning Methods:

   • Previous work [1] has demonstrated that diffusion-based control outperforms RL and imitation learning in controlling high-dimensional nonlinear systems due to the superior capacity of diffusion models in modeling complex system dynamics [2,3]. Our CL-DiffPhyCon extends this advantage by incorporating closed-loop inference, achieving significant gains in efficiency and performance.

3. Classical control methods such as PID and MPC:

   • Classical methods face considerable challenges in controlling high-dimensional, complex systems. For instance, as a common single-input single-output (SISO) controller, the PID is difficult to directly applied to multiple-input multiple-output (MIMO) systems like our 2D incompressible fluid control task (the application of PID to the MIMO system often requires additional decoupling and planning modules, and the control effect of PID is greatly influenced by decouplers and planners, making it difficult to compare fairly). MPC depends heavily on precise models, which are difficult to obtain and often inaccurate for complex physical systems. In contrast, our method is flexible and excels at handling high-dimensional, complex systems without relying on precise models, making it particularly effective for scenarios where system dynamics are implicit or evolving and where classical control methods fall short.

   We have emphasized these differences more clearly in Section 2 "Related Work" in the revised manuscript.

[1] Long Wei et al. “DiffPhyCon: A Generative Approach to Control Complex Physical Systems”. NeurIPS 2024.

[2] Aliaksandra Shysheya et al. "On conditional diffusion models for PDE simulations." NeurIPS 2024.

[3] Jiahe Huang et al. "DiffusionPDE: Generative PDE-Solving under Partial Observation." NeurIPS 2024.

Comment 2: The contributions are unclear. A careful rewriting of the manuscript seems necessary to clarify contributions, and to appreciate the differences between the propose method and the existing ones.

Answer: Thanks for your comment. Contributions were summarized in the last paragraph of Section 1 "Introduction" of our original submission. For clarity, we highlight them below with additional details:

Contributions:

1. A novel closed-loop diffusion control framework (CL-DiffPhyCon):

   We propose a closed-loop diffusion-based control framework that directly interacts with the environment to generate adaptive, real-time control actions. The primary advantage of this framework is its significant improvement in computational efficiency, marking important progress in the field of diffusion-based control. Notably, reviewer sLTC recognized that "This idea addresses one of the main shortcomings of controllers based on diffusion policies."

   Technically, we introduce a novel asynchronous denoising process, enabling closed-loop operation while reducing computational complexity. The proposed asynchronous diffusion model has three notable properties:

   • Efficient asynchronous denoising: The asynchronous diffusion model is a novel denoising model that enables parallel denoising across the direction of physical time steps, allowing earlier decision steps to be sampled sooner. By incorporating feedback into this asynchronous denoising process, closed-loop control is achieved, and redundant sampling steps, as in previous diffusion models, have been eliminated.
   • Theoretical derivation, not heuristic: Our model is derived from a rigorous theoretical analysis of the target distribution. We decompose the complex joint distribution into a product of conditional distributions, which results in only two types of distributions to learn. This theoretical framework may inspire future research on analyzing such complex distributions.
   • Independent effect of acceleration beyond existing fast sampling methods: Our method already significantly improves control efficiency compared to prior diffusion-based methods and can be further accelerated by incorporating fast sampling techniques, such as DDIM. This demonstrates that our method offers an independent and substantial acceleration effect beyond the current fast sampling techniques in the diffusion model community.

2. Significant Performance on Complex Physical Systems:

   • We validate our method on complex physical systems, such as the 1D Burgers' equation and 2D fluid control, showing superior performance with significant sampling acceleration—even in challenging scenarios like partial observations (Table 1), new noise type experiments (Table 1), stochastic control actions (Table 2), and out-of-distribution environments (Table 2 and Figure 3). These results underscore the practical effectiveness of our approach for systems with complex, evolving dynamics.

   We have modified the last paragraph of Section 1 "Introduction" in the revised manuscript according to the above discussion.

(Continued in next piece of response)

We thank the reviewer for the detailed feedback and helpful suggestions. We appreciate the reviewer for noting that our objective of designing closed-loop diffusion-based control algorithms is interesting and timely. The reviewer also highlights the importance of addressing computational complexity of diffusion-based methods. Below, we address the reviewer's concerns one by one.

Comment 1: The paper is poorly presented. This starts with the definition of the problem (where what is known/unknown/available to the designer is unclear, where there is confusion between system and environment, where the stated problem is inherently open-loop and the feedback structure is not evident). As a side note, the problem described in equation (1) is inherently an open loop problem, since no structure is imposed on the input w. In fact, the set of open-loop sequences include the set of closed-loop sequences, but the converse may not be true. Since the system dynamics are deterministic, it is also unclear why using a closed-loop strategy would be necessary.

Answer: Thank you for your feedback. To reduce ambiguity, we replace the keyword "environment" to "system dynamics", still denoted by $G$, which represents the transition of states under external control at each time step, i.e., $u_{\tau+1}=G(u_{\tau}, w_{\tau+1})$ if deterministic, or $u_{\tau+1}$ follows a distribution $p(u_{\tau+1}|u_{\tau}, w_{\tau+1})$ if stochastic. Below, we clarify potential misunderstandings regarding the problem definition:

1. Known and Unknown Aspects of the Problem:

   The designer has access to the following:

   • Initial state ($u_0$): The initial condition that sets the basis for the control process, which is known to the designer.
   • System dynamics ($G$): It represents the transition of states over time in the system, typically determined by implicit PDEs. The evolution of states can only be observed through state measurement. The dynamics could be deterministic or stochastic.
   • Control objective ($\mathcal{J}$): It is defined as a function of the state trajectory ($\{u_1, \ldots, u_N\}$) and the control sequences ($\{w_1, \ldots, w_N\}$), and it drives the optimization of control decisions over physical time steps from $\tau = 1$ to $\tau = N$. $\mathcal{J}$ is specified by the designer.

   The above three components, i.e., initial state, system dynamics, and control objective define a complex physical systems control problem, as clarified at the beginning of Section 3.1 of the revised manuscript.

2. Clarification on Open-Loop vs. Closed-Loop:

   Open-loop control involves predetermined actions without feedback, making it suitable only for systems with simple or well-known dynamics. Closed-loop control, by contrast, adapts based on real-time feedback, which is crucial for complex or evolving systems. Since the evolution of states can only be observed through measurement of the state, the problem in Equation (1) is inherently closed-loop. Namely, each control action depends on real-time feedback, making it impossible to pre-compute all control signals simultaneously, thereby necessitating a closed-loop strategy, with the feedback structure at each time step $u_{\tau+1}$ as provided by the environment.

   To clarify these points, we have revised the manuscript:

   • In Section 3.1, a more explicit definition of what information is known and what is unknown to the designer. In particular, we emphasized the property of system dynamics and the closed-loop nature of our optimization problem shown in Equation (1).
   • In Figure 1 and Figure 2, we replace "environment" with "system dynamics".

Comment 4: Additionally, and importantly, the paper doesn't seem to offer performance guarantees for the proposed method, which makes it difficult to advocate for the proposed method in any practical use. A characterization of the performance of the proposed method is necessary, otherwise the method takes the form of a heuristic that may not be desirable in controls applications.

Answer: Thank you for your comment. We would like to clarify that our method is not heuristic. The use of synchronous and asynchronous diffusion models is theoretically derived from Theorem 1, and the role of feedback in the control process is explained in Proposition 2 of our original submission. Specifically, our original submission provides theoretical support for the following claims:

• Our modified diffusion models can accurately learn and sample from the distribution of control sequences, providing true dynamics for control through guidance sampling.
• Integrating feedback into the asynchronous denoising process has a qualitative impact on the control sequences.

To optimize the control objective during inference, we utilize guidance sampling for action search, as originally proposed in the diffusion model literature [1, 2]. According to these works, guidance sampling is proven to be able to achieve conditional generation, allowing the sampling results to optimize specific objectives. Practically, this process involves applying gradient descent to diffusion model sampling, as demonstrated in Equations (5) and (11) of our original submission. However, while guidance sampling is theoretically validated for conditional sampling and has shown empirical success across multiple domains, establishing a formal theoretical bound for its exact optimization performance remains an open problem in the diffusion model community. We hope future research will provide further theoretical insights, but it is beyond the scope of the present work. We have pointed out this as a limitation of our work in Section 6 "Conclusion and Limitation" of the revised manuscript.

Besides, the challenge in providing performance guarantees also stems from the inherent complexity of the physical systems under control, including non-linearity and stochasticity. Similar difficulties are faced by other control methods, such as PID, which, despite lacking formal guarantees, remain among the most widely used approaches in engineering.

Empirically, we have conducted extensive experiments, including those presented in the original submission and additional results during the rebuttal, which demonstrate the capability of our method in handling complex control tasks effectively. These results highlight the potential applicability of our approach in real-world control settings, even in the absence of formal performance guarantees.

[1] Yang Song et al. "Score-Based Generative Modeling through Stochastic Differential Equations", ICRL 2021.

[2] Hyungjin Chung et al. "Diffusion Posterior Sampling for General Noisy Inverse Problems", ICLR 2023.

Comment 5: The used notation (w for control inputs and u for the system state) is very unusual and, to the best of my knowledge, never been used in any controls paper.

Answer: Thank you for your feedback regarding the notations used in our manuscript. To ensure clarity, we have defined all notations in Section 3 before detailing our proposed method in Section 4 of our original submission. We believe this introduction to notations helps mitigate potential confusion and allows readers from diverse backgrounds to follow the content easily. We understand that selecting appropriate notations requires careful consideration, particularly in such interdisciplinary work that spans multiple research communities—physical systems (PDEs), control theory, and diffusion models—each of which has its own notational conventions. Our primary principle of choosing notations is to highlight clearly the differences from existing methods. Since our work builds on and extends the recent study [1], we chose to use consistent notations, specifically $u$ for state and $w$ for control, to facilitate continuity and ease of comparison. Additionally, in the PDE control community [2,3], $u$ is commonly used for state variables, which further supports our choice.

[1] Long Wei et al. “DiffPhyCon: A Generative Approach to Control Complex Physical Systems”. NeurIPS 2024.

[2] Rakhoon Hwang et al. "Solving pde-constrained control problems using operator learning". AAAI 2022.

[3] Philipp Holl et al. "Learning to Control PDEs with Differentiable Physics." ICLR 2020.

Comment: Would it be possible to obtain guarantees by considering simpler, possibly linear, system dynamics? I'm unsure what the role of PDE dynamics is, since the paper doesn't enter into any details regarding the system under consideration. Using a simpler setting would help to clarify the proposed control method, which is otherwise obscured by the complexities of the setting that are tangential to the proposed method.

Answer: (Continued from previous piece of response)

Lemma 2 If the dynamics $G$ is deterministic, when following the control sequence $z_{1:N} = (w_1(0), u_1(0), \cdots, w_N(0), u_N(0))$ sampled from the synchronous and asynchronous diffusion models, we get the states $(u_{\text{env}, 1}, u_{\text{env}, 2}, \cdots, u_{\text{env}, N})$ satisfying $u_{\text{env}, \tau} = u_\tau (0)$ for every $\tau$.

Prove by contradiction.
If we get another sequence $u_{\text{env}, 1:N} \neq u_{1:N}$, let $k$ be the first index that $u_{\text{env}, k} \neq u_k(0)$ and $u_{\text{env}, i} \neq u_i(0)$ for $i < k$.
According to our paper, the synchronous and asynchronous diffusion models can generate samples from the truthful distribution of the system. Since we get the sample $z_{1:N}$, we know that the system distribution allows $p(u_k(0)|u_{k-1}(0), w_k(0)) > 0$.
Since we get the states $u_{\text{env}, 1:N}$ from the system using $w_{1:N}$, we know that the system distribution also allows $p(u_{\text{env}, k}|u_{\text{env}, k-1}, w_k(0)) > 0$. But $u_{\text{env}, k} \neq u_{k}(0)$ and $u_{\text{env}, k-1} = u_{k-1}(0)$, which is contradictary with $G$ is deterministic. We complete the proof.

Main conclusion If the dynamics $G$ is deterministic, by guided sampling, we can get the approximately optimal control sequence and states.

Proof.
Just combine the Lemma 1 and Lemma 2.

Comment: We thank the Authors for their careful response. However, we remain confused by this manuscript, and particularly by the control problem it claims to address. How does noise enter the dynamics G? The system dynamics do not show any noise and it's unclear how this happens. What are the property of the noise process? It is difficult to accept that the proposed method can solve any optimal control problem for any kind of deterministic or stochastic dynamical system with any kind of noise. Does the paper offer guarantees on the optimality of the resulting control policy? Are the dynamics G stable and is the resulting closed-loop system stable? Since the control input is itself stochastic (sampled from a distribution conditioned on the current stat), how is stability and performance measured and guaranteed?

Answer: (Continued from previous piece of response)

For the question "Since the control input is itself stochastic (sampled from a distribution conditioned on the current state), how is stability and performance measured and guaranteed?", our answer is: although the control input sampled by the diffusion models is stochastic, it does not affect the control performance as the guided sampling optimizes the control objective. Under ideal assumptions, the sampled control signals can be approximately optimal at an arbitrarily high probability. Please refer to the next response for details.

Regarding the comment that "It is difficult to accept that the proposed method can solve any optimal control problem for any kind of deterministic or stochastic dynamical system with any kind of noise," we would like to offer the following clarifications.

Lots of diffusion-based control methods have been proposed and show outstanding performance in control problems like robotic control and flow control [2-5], although they did not claim they can solve any optimal control problem for any kind of system with any kind of noise. Our proposed method moves forward and overcomes the limitation of previous diffusion-based control methods: the very long process of denoising during sampling, which is currently a major obstacle to the wider adoption of such control methods (as pointed out by the Reviewer sLTC). Through extensive experiments, we have observed consistent and impressive control performance improvements. Moreover, we present a guarantee under ideal assumptions for the deterministic system; please refer to the next response for details. Therefore, although we do not claim to meet the reviewer's high standard of solving all possible optimal control problems under all conditions, we firmly believe that our method offers a valuable contribution to existing control methods.

We hope this explanation helps to address the reviewer's concern and provides a more comprehensive understanding of the capabilities of our proposed approach.

[2] Siyuan Zhou et al. "Adaptive online replanning with diffusion models". NeurIPS 2024.

[3] Cheng Chi, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." The International Journal of Robotics Research (2023).

[4] Long Wei et al. "DiffPhyCon: A Generative Approach to Control Complex Physical Systems". NeurIPS 2024.

[5] Anurag Ajay et al. "Is Conditional Generative Modeling all you need for Decision Making?". ICLR 2023.

Comment: Would it be possible to obtain guarantees by considering simpler, possibly linear, system dynamics? I'm unsure what the role of PDE dynamics is, since the paper doesn't enter into any details regarding the system under consideration. Using a simpler setting would help to clarify the proposed control method, which is otherwise obscured by the complexities of the setting that are tangential to the proposed method.

Answer: As we mentioned in the previous response, presenting a guarantee for the practical guided sampling is very challenging and beyond the scope of our work. Therefore, we will make some ideal assumptions about the guided sampling here, only focusing on the insightful part. We list these ideal assumptions as follows:

1. We assume that we can get the ideal classifier mentioned in the paper [1];
2. We don't consider the practical discretization of the SDE in the sampling;
3. We assume the trained diffusion models have learned the exact distribution of the system.

In our practical algorithm, we use DPS[2] to estimate the classifier in the guidance, and the loss function is the greedy version. All the effort here is to illuminate how our method works in the very ideal theoretical setting. We use Lemma 1 to show that the ideal guided sampling can get the approximately optimal sample. And we use another Lemma 2 to show that the sampled sequence will reflect the true feedback states in the system when the system is deterministic. So at least in the ideal setting, our method has the performance guarantee when the system is deterministic.

Lemma 1 By guided sampling, $\forall \varepsilon > 0, \forall \delta > 0$, we can get the approximately optimal sample $z_{1:N}(0)$, s.t., J(u_0, z_{1:N}(0)) - J_\min \leq \varepsilon with probability $1 - \delta$.

Proof.
Given the initial state $u_0$, define an auxiliary random variable $y \in \\{0, 1\\}$, by p(y=1|z_{1:N}(0)) = \exp\\{- \lambda (J(u_0, z_{1:N}(0)) - J_\min)\\}. According to [1], if we use guided sampling, we can get $z_{1:N}(0) \sim p(z_{1:N}(0)|y=1)$. We consider a target set Q(\varepsilon) = \\{ z_{1:N}(0) | J(u_0, z_{1:N}(0)) - J_\min \leq \varepsilon \\}.
Using the guided sampling, we will concentrate the probability into the target set. Specifically,

$

\begin{aligned} P(z_{1:N}(0) \in Q(\varepsilon)) &= \int_{Q(\varepsilon)} p(z_{1:N}(0)|y=1) {\rm d} z_{1:N}(0) \\ &= \int_{Q(\varepsilon)} \frac{p(y=1|z_{1:N}(0))}{p(y=1)} p(z_{1:N}(0)) {\rm d} z_{1:N}(0) \\ &= \frac{E_{z_{1:N}(0) \sim p(z_{1:N}(0)|u_0)} [1_{Q(\varepsilon)} p(y=1|z_{1:N}(0))]}{E_{z_{1:N}(0) \sim p(z_{1:N}(0)|u_0)} [p(y=1|z_{1:N}(0))]} \\ &= \frac{E_{z_{1:N}(0) \sim p(z_{1:N}(0)|u_0)} [1_{Q(\varepsilon)} \exp\{- \lambda (J(u_0, z_{1:N}(0)) - J_\min)\}]}{E_{z_{1:N}(0) \sim p(z_{1:N}(0)|u_0)} [\exp\{- \lambda (J(u_0, z_{1:N}(0)) - J_\min)\}]} \end{aligned}

$

Note that \lim_{\lambda \to \infty} \exp\\{- \lambda (J(u_0, z_{1:N}(0)) - J_\min) = 1_{Q(0)} and $1_{Q(\varepsilon)} 1_{Q(0)} = 1_{Q(0)}$ for any $\varepsilon > 0$. So, we have

$

\begin{aligned} P(z_{1:N}(0) \in Q(\varepsilon)) &= \frac{E_{z_{1:N}(0) \sim p(z_{1:N}(0)|u_0)} [1_{Q(\varepsilon)} \exp\{- \lambda (J(u_0, z_{1:N}(0)) - J_\min)\}]}{E_{z_{1:N}(0) \sim p(z_{1:N}(0)|u_0)} [\exp\{- \lambda (J(u_0, z_{1:N}(0)) - J_\min)\}]} \\ &\to \frac{E_{z_{1:N}(0) \sim p(z_{1:N}(0)|u_0)} [1_{Q(\varepsilon)} 1_{Q(0)}]}{E_{z_{1:N}(0) \sim p(z_{1:N}(0)|u_0)} [1_{Q(0)}]} = 1, \end{aligned}

$

as $\lambda\to\infty$. For any $\varepsilon > 0$, we can make the successful probability arbitrarily close to $1$ by setting $\lambda$ large enough. We complete the proof.

[1] Yang Song et al. "Score-Based Generative Modeling through Stochastic Differential Equations", ICRL 2021.

[2] Hyungjin Chung et al. "Diffusion Posterior Sampling for General Noisy Inverse Problems", ICLR 2023.Just combine the Lemma 1 and Lemma 2.

(Continued in next piece of response)

Comment: I respect the Authors' opinion of the presentation of their work, as well as their efforts in clarifying the concerns raised by the reviewers. However, I still find the paper very hard to read and appreciate. Perhaps the paper is tailored to experts in the field and those that already know the related and cited works, but I strongly doubt that anyone else can appreciate this contribution.

Answer: We understand your concern about the presentation. However, as we clarified previously, the presentation of our method is primarily based on an understanding of diffusion models. Given that diffusion models have become one of the most popular methods for AI-generated content (AIGC), they are increasingly becoming common knowledge in both academic and industrial AI communities. Beyond their successful applications in image [1], video [2], and text [3] generation, diffusion models are also experiencing huge prosperity in fields such as weather forecasting [4], molecule and material design [5], drug discovery [6], and decision-making tasks like motion planning [7] and robot control [8]. Many researchers from diverse backgrounds, including physics, biology, chemistry, materials science, medicine, robotics, and control engineering, are now familiar with this tool. Therefore, our work is aimed at a broad audience with a background in diffusion models, rather than being limited to a small number of experts.

Even in the fields of control and robotics, hundreds of papers on diffusion model-based control have been published in recent years, making it a highly active and rapidly advancing research area. Diffusion models have a unique advantage in solving high-dimensional problems due to their strong generalization capabilities. As a top conference in AI and related fields, ICLR aims at publishing cutting-edge and pioneering research, especially innovative studies that differ significantly from traditional approaches. Such work is more likely to lead to disruptive technological breakthroughs.

[1] William Peebles et al. "Scalable diffusion models with transformers". CVPR 2023.

[2] Duygu Ceylan et al. "Pix2video: Video editing using image diffusion". CVPR 2023.

[3] Tong Wu et al. "Ar-diffusion: Auto-regressive diffusion model for text generation". NeurIPS 2023.

[4] Lei Chen et al. "Swinrdm: integrate swinrnn with diffusion model towards high-resolution and high-quality weather forecasting". AAAI 2023.

[5] Jan-Hendrik Bastek et al. "Inverse design of nonlinear mechanical metamaterials via video denoising diffusion models". Nature Machine Intelligence (2023).

[6] Denise B. Catacutan et al. "Machine learning in preclinical drug discovery". Nature Chemical Biology (2024).

[7] Anurag Ajay et al. "Is Conditional Generative Modeling all you need for Decision Making?". ICLR 2023.

[8] Siyuan Zhou et al. "Adaptive online replanning with diffusion models". NeurIPS 2024.

Comment: We thank the Authors for their careful response. However, we remain confused by this manuscript, and particularly by the control problem it claims to address. How does noise enter the dynamics G? The system dynamics do not show any noise and it's unclear how this happens. What are the property of the noise process? It is difficult to accept that the proposed method can solve any optimal control problem for any kind of deterministic or stochastic dynamical system with any kind of noise. Does the paper offer guarantees on the optimality of the resulting control policy? Are the dynamics G stable and is the resulting closed-loop system stable? Since the control input is itself stochastic (sampled from a distribution conditioned on the current stat), how is stability and performance measured and guaranteed?

Answer: Thanks for your feedback. We agree that in the real-world practice, the system often encounters the issue of noise. As presented in the revised manuscript (Section 3.1), we assume that the system dynamics $G$ has the following form $u_{\tau + 1} = G(u_{\tau}, w_{\tau + 1}, \xi_{\tau + 1})$, where $\{\xi_{\tau + 1}\}$ are independent random variables. When $G$ is stochastic, $\{\xi_{\tau + 1}\}$ are nonzero. In our revised manuscript, we have considered three kinds of noise: system, measurement, and control noise, as suggested by Reviewer GNga.

In 1D experiment, the deterministic Burgers' equation system has the following form: $G_0: \frac{\partial u}{\partial \tau} = -u\cdot \frac{\partial u}{\partial x} + \nu \frac{\partial^2u}{\partial x^2} + w(\tau,x)$, where $u$ denotes the state variable and $w$ represents the control. Then, we consider the system and measurement noise:

• For system noise, we follow the setting in previous work [1], which perturbs the system by Gaussian noise with standard deviation $\sigma=0.025$. The augmented dynamics with additive Gaussian noise have the following form: $G_1: \frac{\partial u}{\partial \tau} = -u\cdot \frac{\partial u}{\partial x} + \nu \frac{\partial^2u}{\partial x^2} + w(\tau,x) +\xi_\tau, \xi_\tau\sim\mathcal{N}(0,\sigma)$. During numerical simulation, the perturbed system $(G_1)$ supersedes the nominal system $(G_0)$ when implementing the control input $w$. Consequently, the feedback state $u$ is obtained as the solution to the stochastic PDE $(G_1)$.
• For measurement noise, it evaluates the case where sensors have noise. The feedback state is characterized by the superposition of the nominal solution $u$ to system $(G_0)$ and a Gaussian noise term $\xi_\tau\sim\mathcal{N}(0,\sigma)$ ($\sigma=0.025$ in the experiments), thereby incorporating measurement noise into the feedback loop while preserving the dynamics of the nominal system.

Please note that our method does not rely on any numerical simulation and can be applied in practice, and the purpose of noise cases is to simulate real-world scenarios. We have added the description of noise in Appendix D.2 of the revised manuscript.

In 2D experiments, we consider the control noise, which means stochastic perturbation during the execution of the control signal in each time step, following the setting in previous work [2]. Specifically, as detailed in Section 5.3 of the original submission, for each trajectory of length $N=64$, the control signal in each physical time step is executed with probability $p=0.1$ as a random action, where the horizontal and vertical components following the uniform distributions bounded by the minimal and maximal values of horizontal and vertical control signals in the training dataset, respectively.

The results in Table 1 and Table 2 in the revised manuscript demonstrate the superiority and robustness of our approach against different types of noise.

[1] Cagatay Yildiz et al. "Continuous-time model-based reinforcement learning." ICML 2021

[2] Siyuan Zhou et al. "Adaptive online replanning with diffusion models". NeurIPS 2024.

(Continued in next piece of response)

We appreciate the high recognition of the reviewer for our work. We are grateful that the reviewer acknowledges our method's significant contribution to addressing the low control efficiency of diffusion control methods. Additionally, we value the insight that our approach can be applied to highly challenging control tasks. Below, we address the reviewer's questions one by one.

Comment 1: I believe the proposed method is excellent and clearly applicable to very difficult control problems involving PDEs. However, most control problems are not of this kind. Why not show the performance on a simpler control problem with a handful of variables, hopefully not requiring high-end GPUs? This could potentially increase the applicability and appeal of the method significantly.

Answer: Thank you for your positive feedback and recognition of our work!

To demonstrate the applicability of our method to lower-dimensional tasks, we performed a new experiment on the control of the inverted double pendulum on a cart. The system consists of two point masses fixed on the end of two massless rigid rods, which are mutually connected and connected to a cart with frictionless hinges. The objective is to control the double pendulum from falling down by exerting a force on the cart. A detailed description and visual illustration of the problem setting can be found in Appendix F.1 and F.2.

We would like to note that this system consists of 6 state variables and 1 action variable, which can be more efficiently simulated. In our implementation, the environment can be simulated with a CPU in real-time. Although the control signal and system states are relatively low-dimensional, we note the control task is non-trivial. Since no torque can be exerted on the hinges, the system can exhibit limited controllability. This system has been studied in the control research community [1,2].

We evaluated the performance of our CL-DiffPhyCon under 3 different levels of random perturbation on the system state and compared the performance with different baselines. Results averaged over 5 seeded runs are reported to enhance soundness. The results of this experiment clearly demonstrate the effectiveness of our CL-DiffPhyCon in controlling this task with improved control efficiency, as can be found in Appendix F.3.

[1] Michael Hesse et al. "A reinforcement learning strategy for the swing-up of the double pendulum on a cart." Procedia Manufacturing (2018).

[2] Masaki Yamakita et al. "Swing up control of a double pendulum." ACC 1993.

Comment 2: Some minor typos: P. 3, L. 112: "It" -> "They" P. 3, L. 140: "has" -> "have" P. 7, L. 376: "well well" -> "perform well"? P. 10, L. 539: "done" -> "drone"? P. 19, L. 1009: "128 grids" -> "128 cells".

Answer : Thanks. All these typos have been fixed in the revised manuscript.

Comment 4: In both examples, the controls are defined as a spatiotemporal function in the PDE model, which is too ideal to be implemented in a realistic system. For PDE system control, typically, the control is implemented as a temporal function. The corresponding actuator is located in the boundary or a small part of the entire spatial domain. It would be interesting to see the results in these more realistic settings by using the proposed method.

Answer: (Continued in next piece of response)

• Thirdly, we also conducted new experiments on a challenging control problem of the inverted double pendulum on a cart, which is also a chaotic system. The system consists of two point masses fixed on the end of two massless rigid rods, which are mutually connected and connected to a cart with frictionless hinges. The objective is to control the double pendulum from falling down. A detailed description and visual illustration of the problem setting can be found in Appendix F.1 and F.2 of the revised manuscript.

  This setting is more realistic in terms of its partial controllability. The control is restricted to the cart, reflecting the spatially limited control in realistic systems. Besides, only the force exerted on the cart can be directly controlled instead of the velocity to reflect a more realistic control setting. This system has been studied in the control research community [3, 4].

  We evaluated the performance of CL-DiffPhyCon by comparing its control performance to different diffusion control baselines under 3 different levels of random perturbation on the state. Results averaged over 5 seeded runs are reported to enhance soundness. They clearly demonstrate the effectiveness of our CL-DiffPhyCon in controlling this task with improved control efficiency, as can be found in Appendix F.3 of the revised manuscript.

[3] Michael Hesse et al. "A reinforcement learning strategy for the swing-up of the double pendulum on a cart." Procedia Manufacturing (2018).

[4] Masaki Yamakita et al. "Swing up control of a double pendulum." ACC 1993.

Comment 5: In the Burgers’ equation example, there is no difference between the full observation and partial observation cases, because the latter case only observes and controls two parts of the entire domain. In the case considered in the manuscript, one can simply treat each observed sub-domain as a total entire domain and apply the algorithm on the PDE with new spatial coordinates/ranges/boundary conditions. Partial observation typically refers to the case using partial observation to control the entire domain. The control performance results of the unobserved state should be added to fully demonstrate the effectiveness of the proposed algorithm. Similarly. practical applications often have sensors installed in the boundary of PDE systems. It would be good to also add results by using boundary observation data.

Answer: Thanks for your question. We conducted new experiments to show the control capacity on partial observation. In this setting, sensors only measure 1/8 of the spatial domain (16 sensors are evenly placed to partially observe). The POPC results are shown in Table 4 of the previous response (Official Response to Reviewer GNga (4)). We can see that performance slightly decreases due to the reduction of observation, but our method still has a significant improvement over the baselines.

We have added the experiments in Table 1 of Section 5.2, of the revised manuscript.

Comment 6: The limitations of the proposed method should be moved to the conclusion section for better readability.

Answer: Thank you for the suggestion. In the revised manuscript, we have moved the discussion of limitations from the appendix to Section 6 "Conclusion and Limitation".

Comment 4: In both examples, the controls are defined as a spatiotemporal function in the PDE model, which is too ideal to be implemented in a realistic system. For PDE system control, typically, the control is implemented as a temporal function. The corresponding actuator is located in the boundary or a small part of the entire spatial domain. It would be interesting to see the results in these more realistic settings by using the proposed method.

Answer: Thanks for your question. The settings of our experiments follow previous works [1,2], where the control actions are the spatiotemporal functions. According to your suggestions, we have conducted new experiments when control actuators are located in a small part of the spatial domain, both in 1D and 2D experiments.

• Firstly, in 1D experiments, we reduce the dimension of control, and actuators are evenly placed in 1/8 spatial domain (16 actuators are placed to partially control)(FOPC). The illustration of placement is shown in Appendix D.2 of the revised manuscript. The results are evaluated on the entire spatial domain, which is shown in the following Table 4. Average inference time is evaluated using a single NVIDIA A100 80GB GPU with 16 CPU cores.

  We can see that the reduced controllable range generally diminishes the effectiveness of diffusion-based control methods, yet our method consistently performs better than the baselines. BC, BPPO, and PID demonstrate improved performance because only a small part of actions need to be optimized, making it easier to fit compared to high-dimensional actions, while they still show significantly lower performance compared to diffusion-based methods. We have added the results and analysis in Section 5.2 of the revised manuscript. We also considered partial observation (POPC). (As a common SISO controller, the PID is difficult to directly apply to MIMO systems. Its application to the MIMO system often requires additional decoupling and planning modules. Thus, we only apply the PID in 16 sensors for observation and 16 actuators for control (POPC).)

Table 4: Comparison results on 1D Burgers' equation control under FOPC and POPC settings.

[1] Rakhoon Hwang et al. "Solving pde-constrained control problems using operator learning". AAAI 2022.

[2] Philipp Holl et al. "Learning to Control PDEs with Differentiable Physics." ICLR 2020.

(Continued in next piece of response)

We thank the reviewer for the valuable and detailed feedback. We are glad that the reviewer recognizes the clarity and experimental strengths of our work. Below, we address the reviewer's questions one by one.

Comment 1: How restrictive is the consideration of noise-free control? There are many noise issues arising from complex engineering systems and applications. Can the proposed method be extended to account for plant, control, and measurement noise?

Answer: Thanks for your question.

• Firstly, in the 2D incompressible fluid control task in the original submission, we have added noise in the execution of control, following previous work [1]. Specifically, as detailed in Section 5.3 of the original submission, for each trajectory of length $N = 64$, the control signal in each physical time step is executed with probability $p = 0.1$ as a random noise, where the horizontal and vertical components following the uniform distributions bounded by the minimal and maximal values of horizontal and vertical control signals in the training dataset, respectively. The results reported in Table 2 of the original submission demonstrated the superiority and robustness of our approach against control noise.

  To further evaluate the control performance with respect to control noise, we conducted new experiments on a new boundary control setting, where control signals are restricted to only 8 cells for each of the 4 side exits of the $64\times 64$ fluid field. Consistent with our original setting (renamed as "large domain control" in the revised manuscript), the control signal in each physical time step is still executed with probability $p = 0.1$ as a random noise. Evaluations of different methods are conducted on both fixed map and random map modes. The results are shown in the following Table 1. The average inference time of two evaluation modes on a single NVIDIA A6000 48GB GPU with 16 CPU cores is also reported.

Table 1: Comparison results on 2D incompressible fluid control under control noise with probability 0.1 in the additional boundary control setting.

These results indicate that even under the perturbation of control noise, our method still achieves acceptable control performance under this more challenging boundary control setting with a much smaller number of actuators. Compared with baselines, our CL-DiffPhyCon achieves the best control performance. The fast sampling DDIM method could still be applied to our method for significant acceleration of control.

We have added the above results and analysis in Section 5.3 of the revised manuscript, with further details presented in Appendix E.

[1] Siyuan Zhou et al. "Adaptive online replanning with diffusion models". NeurIPS 2024.

(Continued in next piece of response)

Thank you for your recognition and recommendation for our work. We are pleased that our response has successfully addressed all your comments.

Moreover, we gently remind you that in ICLR's scoring system, a score of 6 indicates a "marginally above the acceptance threshold", while a score of 8 represents "accept". If you believe our work merits acceptance, we hope you might consider giving us an "accept", as it would greatly support our work.