PRDP: Progressively Refined Differentiable Physics

Dear reviewers,

Thank you for your valuable inputs. We focussed our efforts towards clarity and accessibility, and we are glad for your feedback suggesting opportunities for better presentation. Since our work combines ideas from two slightly disconnected domains, i.e., differentiable physics and bilevel optimization, we acknowledge that certain domain-specific terminology and notation may not be naturally evident. Below we address specific concerns raised in your review.

1. Pseudo Code: Under Algorithm 4 in the appendix, we have provided a pseudo code for the PRDP control algorithm from section 3. For better comprehension,
   1. We will create a schematic that explains the basic idea of the algorithm more intuitively.
   2. We will add pseudo-code of a typical solver-in-the-loop training pipeline in the main text that additionally shows where we invoke the PRDP control algorithm and the physics refinement.
2. Where is the neural network used in the global framework/experiments: The global framework is a neural network training pipeline where the gradients pass through an iterative physics solver. Figures 17-19 in the Appendix depict this framework, and Appendix E provides a detailed explanation on how the networks are trained, and how they are employed for inference.
3. What are inner/outer optimization problems in the examples: In all cases, the outer problem is an optimization problem that trains a neural network (or solves an inverse problem in the Poisson case), while the inner problem is the solution to the linear system that represents the physics (ref. Equation 1). These details are available in the appendix E. We value your feedback and will make suitable edits in the main text to present these details more explicitly.
4. Notations: We will add a paragraph summarising all notations in the appendix.
5. Applications to other iterative processes: Indeed, the core mechanisms of PRDP are not bound to just iterative linear solvers. We also argue that our approach is inspired by other fields of bi-level optimization as we discuss in the second paragraph of Section 5. To the best of our knowledge, PRDP is the first time such a scheduling approach is used for contexts of training neural networks with differentiable numerical solvers. Conversely, we are optimistic that the aspects of PRDP will find their way back to the fields of hyperparameter optimization, meta learning, etc. This is especially noteworthy since our core motivation of Pedregosa (2016) works with machine learning models requiring a convex optimization fit. Solving linear systems can also be seen as a (quadratic) convex optimization albeit with sparse and structured system matrices (in the case of discretized PDE models) instead of dense data matrices.
6. Performance difference when trained w/ and w/o PRDP: With PRDP, we reduce training costs without significantly affecting performance. Indeed, there is a very small difference visible in Figure 1. Conversely, we also see a very small improvement in performance (see e.g. figure 4 (b)). These minor differences were usually within bounds of the variance over random seeds. We will share the exact numbers in the revised version. In general, our results show that PRDP accelerates training while retaining the full accuracy at inference time.
7. Why based on Validation Loss: Thank you for this interesting question. Our choice of a validation metric is based on the following observations.
   1. Previous approaches, i.e., Pedregosa (2016), implement progressive refinement through a sequence of tolerances. While this approach provides PR savings, it does not enable IC savings in problems where a certain level of incomplete inner refinement is sufficient for the network’s performance. In our approach, this refinement level is effectively identified by continuously examining a performance metric.
   2. At first guess, one may pick training loss to serve as this performance metric. However, the training loss can be a misleading indicator. We observed in some experiments (e.g. emulator training for the heat equation) that with incompletely converged physics, the network training loss reduces while the validation error plateaued. Consequently, basing PRDP on training loss could result in a network that has low training loss but does not generalise to unseen data. By using validation errors, we make PRDP robust towards sufficiently refining the physics ensuring network performance.

8. Validation values at inference: During inference, PRDP does not play any role. It applies only to training the neural network. In problems where the inference involves not only the neural network but also the physics (e.g. correction learning setups like our Navier Stokes example), the physics is fully converged during inference.

9. Applications of these NNs: The neural emulators are trained to become cheap surrogates for forecasting problems modelled by PDEs. Please cf. our first reply to reviewer 1fLe. We acknowledge that the introduction can be enhanced by providing more details on the bigger picture.

10. Notation:

    1. The subscript $h$ is taken from standard CFD textbooks to express the discretization of a continuous variable, where $h$ stands for the width of a cell in discretized space. Thank you for pointing this out; we will mention this explicitly in the main text.
    2. Yes, $\\theta$ refers to the arguments for the outer optimization problem, i.e., the neural network parameters. The linear system of the physics follows the neural network’s computation (ref. compute graphs in figures 17-19), hence the linear system is indirectly parameterized by $\\theta$.

    We will add a figure in the main text that provides a general overview of our experimental framework, clarifying the interaction of the neural networks and the physics.

11. Navier Stokes = real world?: We used “real-world” synonymous with the difficulty the Navier-Stokes equation usually poses for numerical integration. It is a nonlinear system of equations, with an asymmetric advection characteristic and has a saddle-point structure. While our data is indeed synthetic, the step from our model to engineering CFD simulations (which are used to simulate the real world) is smaller than the step from the illustrative heat emulation to the Navier-Stokes. Hence, the Navier-Stokes example is the hardest test case of our submission.

12. Section 2.2 problem setup:

    1. Yes, it is an inverse problem. We know the response displacement to the Poisson equation, then we model a forcing function with a free parameter and fit this parameter via comparing the predicted Poisson solution with our reference.
    2. No, there is no trajectory because the Poisson equation is steady-state. Hence, the application is finding the forcing term associated with a given steady-state displacement.

13. Performance of the test models: Thank you for this question regarding the bigger picture. For our work, we were purposefully interested in running experiments with neural networks. Our work targets neural emulators/surrogates or neural operators that are trained through differentiable physics. This method has shown success in smaller fluid problems (Kochkov et al. and Um et al.) and most recently in weather and climate (NeuralGCM). In other words, performant models enabled by differentiable physics pipelines are a proven strategy. Despite the promise of neural-hybrid models, they often lack adoption since executing and differentiating over classical numerical solvers during training is costly. The focus of our work is not on improving the trained models, but rather to improve the training methodology.

14. What if the neural network size increases/its expressiveness improves? Are the performances better?: Based on your feedback, we have conducted a scaling test with the neural network parameter size for the heat 1d case. For an order of magnitude increase in the neural network size, the network’s accuracy improved by nearly a quarter of an order of magnitude, while the savings achieved by PRDP was nearly the same (79% and 81%). We will add the details to the revised pdf.

15. Training times:

    1. We provide training times for the challenging Navier Stokes experiment in Figure 1 with a notable 62% improvement.
    2. Solving time: As we pointed out in the previous points, our methodology pertains only to training but not inference.

Dear Reviewer,

Thank you for your thoughtful feedback and for raising your score. We have incorporated your recommendations into the revised manuscript to improve clarity and presentation. Specifically:

• Clarity Enhancements: We added multiple schematics (new Figures 1, 5, and 9) and included pseudo-code (Listing 1) to clarify the integration of the neural network within the framework.
• Nomenclature: A detailed nomenclature section is now included in Appendix A to improve readability.

Below, we address your specific replies:

6. Adding training duration: We have added wall-clock training times for PRDP versus fully refined physics in Figure 24 and Section G.1, and the relevant savings% in the main text.

7. Addressing your points individually:

   a. Ablation Study on Training Loss as a PRDP Performance Indicator: We are compiling the data and will include the corresponding plots in the final revised PDF tomorrow.

   b. Using PRDP at inference: Since there is no outer optimization during inference (the network is already trained), we assume your question refers to leveraging incomplete convergence (IC) savings during inference. Specifically, if PRDP terminates refinement at $K_{\text{max}}$ during training, can this level of refinement be used for inference?

   • This is an intriguing idea, and we have added it to the outlook section.
   • In general, if inference conditions match the validation metric computation (e.g., same initial condition distribution and number of unrolled steps), $K_{\text{max}}$​ may suffice without degrading performance.
   • However, practical inference conditions often differ from training, so reduced refinement could negatively impact generalization. For robust performance, we recommend full refinement during inference.
   • Note that only the last experiment of our manuscript (section 4.4 on neural-hybrid emulators) is a setting that involves a physics solver during inference. The pure prediction neural emulator for Heat and Burgers and the inverse problem for the Poisson equation do not. Hence, there can not be any IC savings during inference since there is no iterative process during inference.

11.We have removed the term “real-world” from the introduction to avoid misunderstandings.

Dear reviewer,

We are glad that you found our answers helpful. Thank you for your vote of acceptance.

We have just uploaded the final revised pdf with the complete set of supplementary results. We briefly summarize them here:

• Network expressiveness and PRDP savings: We conducted an ablation for the experiment on emulating the Heat PDE 1D. For this, we increased the emulator's parameter space by one order of magnitude. This improves the final validation accuracy (due to the increased capabilities of the model) but only marginally affects the PRDP savings. Most importantly, it does not lower the IC savings, they are persistent across the varying network sizes.

• To answer your original question: "For the IC savings ... what if the neural network size increases/its expressiveness improves? Are the performances better?" Our ablation shows that the accuracy of the outputs improves, while the IC and PR savings remain persistently high. Together they reduce the iteration count by 80% across the three network sizes.

• Running PRDP on the training loss: We repeated Heat 2D, Heat 3D, Burgers and Navier-Stokes with both training loss and validation loss as the PRDP indicator. The results show that training loss can also serve as a criterion for stepping in PRDP. The final accuracy achieved of the emulators is similar to the results with validation metrics. Using training losses yields slightly higher PR savings because the refinement is slower: when the validation metric plateaus the training loss often still continues to go down. This is also the observation that underpinned our initial reply above. While training loss as the PRDP indicator worked for our experiments, we recommend caution: We expect that for more complex problems, such as those with multi-modality or spurious minima, the training loss will be less reliable than the validation loss. Moreover, the convergence is smoother if the validation metric is used as a PRDP indicator.

We thank you again for your thoughtful questions.

Dear reviewer,

Thank you for your valuable feedback. We greatly appreciate that you share our intuition that physics solvers do not always need full convergence to achieve optimal network accuracy. Below, we address your remarks and questions in detail:

1. More Background on Differentiable Physics: We acknowledge that the introduction could better emphasise the broader problem domain requiring differentiable physics. Our work fits into the category of neural emulators, surrogates, or neural operators, which aim to enable efficient forecasting for PDE-governed problems across various scientific and engineering domains. Solving PDEs is fundamental to fields ranging from quantum mechanics (Schrödinger Equation) to structural engineering, fluid dynamics, weather forecasting, climate research, and astrophysics.
   While many recent approaches are purely neural, hybrid methods that integrate classical numerical solvers with neural components have demonstrated superior performance. For example, these hybrid models have shown success in small-scale fluid problems (e.g., Kochkov et al., Um et al.) and large-scale systems like weather and climate modelling (e.g., NeuralGCM). Notably, the experimental setup in Section 4.4 is conceptually similar to these prior works.
   Despite their promise, neural-hybrid models face limited adoption due to the computational cost of executing and differentiating through classical solvers during training. Since the majority of compute time in engineering-relevant PDE solvers is spent resolving nonlinear and linear systems, PRDP directly addresses this bottleneck. As we point out in the outlook, PRDP could catalyse broader adoption of differentiable physics. We recognize that this broader context was underdeveloped in the introduction and will revise it to ensure these connections are clear.

2. The role of the neural network: We apologise for not adequately highlighting the role of neural networks in the experimental setups described in Section 4. To clarify, neural networks are utilised in three distinct contexts in our work:

   1. Neural emulator learning (introduced in Section 2.3, and used in Sections 4.2 and 4.3): Here, the neural network is trained to replace a numerical time stepper, i.e., the simulation method advancing a state in time.
   2. Neural correction learning (used in Section 4.4): In this context, the network is trained to correct or modify predictions from a coarse numerical simulator, forming a neural-hybrid emulator.
   3. Poisson inverse problem (introduced in Section 2.2, and used in Sections 4.1): This involves a parameterized right-hand side (RHS). While the RHS in our example is defined by the first three eigenmodes of the Laplace operator scaled by one parameter each, one could alternatively use a neural network to parameterize the RHS in higher-dimensional settings.

   We agree that the main text could benefit from clearer explanations of these contexts and improved visual aids, such as simplified versions of the flowcharts from Figures 17-19. We will make these changes in the final revision.

3. Does PRDP influence network accuracy: PRDP does not negatively affect network accuracy, which remains consistent within the variability introduced by random seeds (for network initialization and stochastic minibatching). Instead, PRDP improves training efficiency by reducing computational costs while preserving accuracy. We discuss this in Section 2.3 (see also Fig. 3b) and confirm it experimentally in Fig. 4.

Dear Reviewer,

Thank you for your thoughtful evaluation of our paper. We are delighted that you found our algorithm straightforward, appreciated the substantial savings, and enjoyed the writing. Below, we address your remarks and questions in detail:

1. Limited technical contribution: A key contribution of our paper lies in demonstrating that PR and IC savings exist in a differentiable physics setting. While we agree that the proposed approach—progressively increasing the number of solver iterations—is simple, we view this simplicity as a strength. It underscores that these savings are accessible without requiring highly complex modifications to existing workflows.
2. Applicability to other physics solvers: This is an excellent point. While we focus on linear and simplified nonlinear systems (e.g., Burgers' and Navier-Stokes equations, solved with a one-step Picard approximation akin to an Oseen problem as per Turek 1999), the extension to fully nonlinear systems is indeed a natural next step. Fully resolving nonlinear systems (e.g., through Newton-Raphson methods) introduces a non-quadratic loss landscape, which may require specific adaptations to PRDP. However, PRDP could potentially be applied to schedule the number of nonlinear solver iterations (e.g., Picard or Newton steps), yielding similar savings. For context, incomplete resolution of nonlinear residuals is a common practice in numerical methods like the PISO algorithm for Navier-Stokes.
   Beyond nonlinear systems, we believe our work already covers a broad range of linear cases, including symmetric positive definite matrices, asymmetric matrices, parameterized matrices, and saddle-point problems. Are there specific solver types or problem classes you had in mind that we did not address?
3. Applicability of the considered examples: We deliberately chose simpler examples to illustrate the key mechanisms behind PR and IC savings. As noted by other reviewers, differentiable physics is a technically challenging domain, and simpler cases provide a clear view of these mechanisms. We aim to clarify in the paper that scaling up to higher resolutions (e.g., transitioning from 2D to 3D, or to larger-scale systems) builds directly on these foundational insights.
4. Sensitivity of the PRDP parameters: For most cases (e.g., 1D/2D Heat and Navier-Stokes), selecting hyperparameters was straightforward, and we experimented with a limited range of values around the defaults specified in the pseudo code (ref. algorithm 1). However, the Burgers case required more extensive tuning. We acknowledge that parameter sensitivity can vary depending on the problem, and we will ensure this aspect is discussed in the final version.
5. PRDP on domains with complex BCs and irregular meshing: We expect PRDP to generalise to such settings. The IC savings are rooted in the phenomena described in Section 2.3 and should persist in more complex configurations. Similarly, PRDP schedules solver refinement efficiently, approaching the necessary $K\_{\\text{max}}$ for convergence.
   However, the applicability of PRDP depends on the success of the underlying differentiable physics process. If a linear system cannot converge due to issues like poorly conditioned matrices (e.g., from stretched meshes or difficult boundary conditions), differentiable physics (and thus PRDP) would also struggle. Conversely, for reasonable meshing and boundary handling, the behaviour of the system matrix should align with our simpler experiments, avoiding worst-case scenarios where refinement is forced to full resolution.
6. Could PRDP work under incomplete physics: PRDP’s performance likely depends on the numerical characteristics of the incomplete physics. If the incomplete physics minimally impact the system matrix spectrum and allow for primal solutions, PRDP should remain effective. Initial stabilisation might require higher $K\_0$ values, slightly reducing PR savings, but IC savings (the dominant contributor) could compensate or even improve in such cases.
7. How does noise in observations affect PRDP: Noise introduces similar challenges as incomplete physics. Higher initial stabilisation costs might marginally impact PR savings, but IC savings could mitigate this. Analogously to geometric vs. algebraic multigrid methods, PRDP operates effectively at the numerical level, and modest noise levels should not undermine its utility.

Dear Reviewer,

Thank you for your constructive feedback on our manuscript. We are pleased that you found PRDP to be an original contribution. Below we address your remarks and questions in detail:

1. The choice of model/experimental complexity and baseline: We chose the set of experiments specifically to illustrate the key aspects of PRDP (progressive refinement and incomplete convergence savings) as well as showcase how PRDP is applicable in different problem scenarios. The experiments thereby show well-behaved symmetric positive definite linear matrices, asymmetric upwinding matrices, parameter-dependent matrices and saddle-point problems. We acknowledge that under the investigated resolutions, a direct solver would generally be preferable. Yet, we think that the savings achievable with PRDP should also exist for higher spatial resolutions (hence larger system matrices). This is because when resolutions of PDE discretizations (on uniform Cartesian grids) grow, sparsity patterns and the form of the spectrum stay almost the same, albeit the condition number grows indicating a slower convergence (-> requiring more linear solver iterations, but PRDP likely will deliver similar percentage savings). We appreciate your feedback and executed a 3D heat emulation experiment (for results, see point 3).
2. Differences in PRDP savings over increasing experimental difficulty: Thank you for raising this interesting observation. Training a neural network for the Burgers setup was particularly challenging. When training was performed using very coarse physics (less than 4 solver iterations), the training severely diverged. We started training with a relatively high level of refinement for this case, hence the lower savings. While divergence may reduce PRDP’s benefits, problems of increasing complexity do not necessarily pose inherent issues. For instance, in the correction learning experiment for Navier Stokes, PRDP enabled nearly the same saving as the heat 1D/2D emulator training experiments. Additionally, we have added a 3D heat emulator learning example as highlighted below.
3. Computational Experiment where an iterative solver is necessary: We have added a three-dimensional heat emulator learning example and can confirm similar savings of 80% (62% IC savings and 18% PR savings). This indicates that similar savings can be expected for larger, more complex problems. We will add the details in the revised pdf.
4. Harder baselines for the linear emulator learning experiment: The Helmholtz equation is a steady-state equation and in its core formulation, it is an eigenvalue problem. Hence, our heat emulator learning problem does not transfer directly. While we can imagine that PRDP might also be applicable when using iterative eigenvalue solvers (such as the power method or the Lanczos algorithm), this is beyond the scope of the rebuttal. A more trivial extension of the existing experiments would be to perform the Poisson inverse problem instead of the Helmholtz equation with an inhomogeneous parameterized forcing term (similar to the Poisson equation being the inhomogeneous extension to the Laplace equation). Ultimately, the Helmholtz equation (when considered of the form -\\Delta u \+ k^2 u \= f) is similar to the Poisson equation (in terms of matrix sparsity pattern and SPD-property). Thus, we think PRDP could potentially also yield benefits in Helmholtz solvers. However, for this rebuttal we have focused on the 3D heat problem, as outlined above.
5. Correction in total savings percentage: Thank you for pointing out this small typo. We will fix the total savings numbers in the pdf.