Understanding Generalization in Physics Informed Models through Affine Variety Dimensions

We greatly appreciate the time and expertise you invested in reviewing our work. Your comments provided important guidance for our revisions.

Q1: Can the authors provide further intuition on how their study applies to more common architectures as neural networks? In principle the approach of solving a least squares problem to find the estimator is similar to the idea behind extreme learning machines (which also exists in the physics-informed version). Can the theory in your article be applied to such a case?

We thank the reviewer for this insightful question.

As the reviewer correctly notes, our theory applies directly to Extreme Learning Machines (ELM) or random feature models. In these cases, the weights of the final layer correspond to the vector $w$ in our analysis.

Extending the theory to standard multi-layer Neural Networks (NNs) is more challenging. In principle, for NNs with polynomial activations (which can approximate smooth activation functions like tanh), one could derive an inequality similar to our Theorem 3.5, as outlined in the proof strategy at the end of this response. However, we identify two significant challenges at this stage:

• The system of equations defining the affine variety $V$ would consist of highly complex, high-degree polynomials. This makes the numerical computation of $d\_V$ computationally prohibitive, thereby hindering both theoretical validation and practical application.

• Recent research indicates that the generalization of over-parameterized models like NNs is influenced not only by the geometry of the hypothesis space but also significantly by the implicit bias of the optimization algorithm. Consequently, a bound like our Theorem 3.5, which solely considers the geometry of the hypothesis space, may exhibit a significant gap from empirically observed performance.

Q2: What is the practical outcome of Theorem 3.5? Can the authors clarify, based on the aforementioned theorem, what should be good practices for building/training physics-informed models?

The primary practical outcome of Theorem 3.5 is the use of the affine variety dimension, $d_V$, as a guiding principle for the efficient selection of basis/trial functions.

For instance, in large spatio-temporal domains where using a dense set of collocation points (a common practice in PINNs) is computationally infeasible, $d_V$ can serve as a metric to guide an effective and tractable placement of these points prior to training.

As a promising future direction, our framework suggests an alternative to methods like weak PINNs where trial functions are learned simultaneously with the main solution. Instead, one could first optimize the set of trial functions by explicitly minimizing $d_V$ before the main training process begins.

Q3: For the case of PINNs, the training is typically performed on a dense set of collocation points, in order to cover properly the whole training domain. How relevant is ensuring good generalization in these scenarios?

Our theory is applicable regardless of whether the set of collocation points is dense or sparse. It provides a formal upper bound on the generalization error and helps determine the amount of training data, $n$, required to guarantee a desired performance level.

As we state on lines 188-194, when the collocation points are sufficiently dense, we expect the information from the differential operator $\mathscr{D}$ to be maximally utilized. In our framework, this corresponds to a configuration that minimizes the intrinsic dimension of the solution space, i.e., $d\_{\\mathcal{V}} = \\min\_{\\mathcal{T}} \\dim(\\mathcal{V}(\\mathscr{D}, \\mathcal{B},\\mathcal{T}))$.

Outline of the Proof Strategy for NNs

The goal is to derive a generalization bound similar to our Theorem 3.5 for the hypothesis space $\mathcal{H}\_L$ of an NN by controlling its Rademacher complexity. While the overall framework is similar to that for linear models, we introduce a key extension to handle the non-linear, hierarchical structure of NNs.

• Step 1: Reduction from Loss Class to Hypothesis Class Following standard learning theory, the generalization error is bounded by the Rademacher complexity of the loss class, $\mathfrak{R}(\ell \circ \mathcal{H}\_L)$. We then apply the Contraction Principle, leveraging the Lipschitz property of the loss function, reducing the problem to bounding the Rademacher complexity of the hypothesis class itself, $\mathfrak{R}(\mathcal{H}\_L)$. This step is common to both linear models and NNs.

• Step 2: Bounding Hypothesis Complexity via Dudley's Integral Next, we bound $\mathfrak{R}(\mathcal{H}\_L)$ using Dudley's integral theorem. This theorem bounds the complexity via an integral over the covering numbers of the hypothesis space, $\mathcal{N}(\mathcal{H}\_L, \varepsilon, \\|\cdot\\|\_{\infty})$. This step represents the key departure from the analysis of linear models.

• Step 3: Bounding the NN's Covering Number Directly evaluating the covering number for a complex function space like that of an NN is often intractable. Our approach is to bound the covering number of the function space by that of the more tractable parameter space.

  • Stability Analysis: Through a recursive, layer-by-layer stability analysis, we relate distances in the parameter space (weights) to distances in the function space. Specifically, we show that if two parameter vectors $w$ and $w'$ are $\varepsilon$-close, their corresponding functions $f\_w$ and $f\_{w'}$ are bounded by $\\|f\_w - f\_{w'}\\|\_{\\infty} \\le \\ell\_L \\varepsilon$. The stability constant $\\ell\_L$ depends on the network's architecture (e.g., depth $L$ and weight norm $R$).
  • Covering Number Relation: This stability analysis yields the critical inequality $\\mathcal{N}(\\mathcal{H}\_L, \\ell\_L \\varepsilon, \\|\\cdot\\|\_{\\infty}) \\le \\mathcal{N}(V\_R, \\varepsilon, \\|\\cdot\\|\_2)$, which bounds the function space's covering number by the parameter space's.

• Step 4: Leveraging the Geometry of the Parameter Space This reduces the problem to bounding the covering number of the parameter set $\mathcal{V}\_R$ by applying the Lemma B.2.

• Step 5: Final Assembly Finally, we substitute the covering number bound from Step 4 into the Dudley integral from Step 2 (accounting for the scaling by the stability constant $\ell_L$). Solving this integral yields a concrete upper bound on $\mathfrak{R}(\mathcal{H}\_L)$ in terms of $d\_V$ and $\ell\_L$. This result is then used in the generalization inequality from Step 1 to complete the proof of the final theorem for PINN (similar results of Theorem 3.5).

Additional Key Assumptions

This extension requires two key assumptions:

• Lipschitz continuity and boundedness of the activation functions.
• A norm constraint on each layer's weight matrix.

We would like to express our sincere gratitude for your time and effort in reviewing our manuscript. Your comments have been very helpful.

Q1: Line 156, what is d ("ambient input dimension")? For example, when we use a 1000-parameter PINN to solve a 2D PDE, is d=2 or d=1000?

$d$ is the number of parameters in the model, not the dimension of the PDE's domain. So in your example, $d=1000$.

Q2: Line 307, the diffusion coefficient is 0?

We apologize for this typo. The correct equation is:

We will correct this in the revised manuscript.

Q3: I'm slightly confused by the goal of PILR. Is it a theoretical framework or a training method? Section 3 seems to frame it as a theory, while Section 5 seems to suggest that it is a training method.

The goal of our paper is to provide a theoretical framework for analyzing the learning problem we refer to as PILR. Section 3 details this theoretical analysis, while the purpose of Section 5 is to empirically validate our theory, not to propose a new training method. In fact, the algorithms used to solve the PILR problem in Section 5 are existing, standard methods: the analytical solution from [16] for linear cases and Adam optimizer for nonlinear cases (see lines 283-287).

To prevent this misunderstanding in the manuscript, we will revise the text to clarify this distinction. The revised sentence at the beginning of the experimental section will read:

Line278: To validate our theoretical analysis, our experiments compare the generalization performance of two fundamental approaches to learning the target function $f^*$: standard Ridge Regression (RR) with the fixed basis $\\mathcal{B}$, which uses only observational data, and Physics-Informed Linear Regression (PILR), which additionally incorporates constraints from physical laws.

Q4: Can you give a simple nonlinear example in action?

Yes. We conducted experiments with two nonlinear equations: the nonlinear Bernoulli equation and the nonlinear diffusion equation in Section 5.

• For the Bernoulli equation, the nonlinear term is $Qy^{\\rho}_{\tau}$, where we set $\rho = 2$.
• For the diffusion equation, the diffusion coefficient depends on the solution: $c(u) = 0.1/(1 + u^2)$

Our experiments confirm that even in these nonlinear cases, the improvement in generalization error is well-explained by the dimension of the resulting affine variety, $d_V$.

Q5. I think that some parts can be dropped without compromising the main idea. For example, I am confused by $\beta$. My best attempt to understand it is that it is related to topology due to the nonlinear differential operators, but otherwise it remains mysterious.

We understand that $\beta$ is an abstract concept, but it is a crucial component of our work.

A key strength of our theory is its applicability to nonlinear differential equations, which makes the topological analysis involving $\beta$ indispensable. To clarify its role, we provide several explanations in the manuscript:

• A concise definition in Lines 159-165.
• A discussion of how $\beta$ appears in the covering number upper bound in Lines 166-177.
• A detailed, intuitive explanation with examples in Appendix B.2.

Here is a brief summary of the intuitive explanation from Appendix B.2:

Consider a circle defined by $V = \\{ (x, y) \in \mathbb{R}^2 \mid x^2 + y^2 - 1 = 0 \\}$. The ambient space dimension is $d = 2$, and the circle's intrinsic dimension is $d_V = 1$. In this 2D plane, we can consider affine subspaces $L$, such as lines (1-dimensional) and points (0-dimensional). The definition of a $(\beta, d_V)$-regular set depends on the codimension of these subspaces, which is $\text{codim}(L) = d - \dim(L)$.

Case 1: Intersection with Lines

For any line $L$ in the plane, its dimension is $\dim(L) = 1$. The codimension is therefore $\text{codim}(L) = 2 - 1 = 1$. Since $\text{codim}(L) \leq d_V$ (as $1 \leq 1$), this falls under the first case of the definition. Here, $\beta$ is defined as the maximum number of path-connected components of the intersection $V \cap L$.

Let's see what happens when we intersect our circle $V$ with a line $L$:

• No intersection: If the line misses the circle, $V \cap L$ is empty (0 components).
• Tangent line: If the line touches the circle at a single point, $V \cap L$ is a single point (1 path-connected component).
• Secant line: If the line cuts through the circle, $V \cap L$ consists of two distinct points. Since these points are separated, this set has two path-connected components.

The maximum number of path-connected components we can get is 2. Therefore, for the circle, $\beta = 2$. Intuitively, $\beta$ is related to the topology of the variety $V$. A line can intersect the single "loop" of a circle at most twice.

Case 2: Intersection with Points

Now, consider a point $L$ in the plane. Its dimension is $\dim(L) = 0$. The codimension is therefore $\text{codim}(L) = 2 - 0 = 2$.

Since $\text{codim}(L) > d_V$ (as $2 > 1$), this falls under the second condition, which requires $V \cap L$ to be empty for "most" such subspaces. This condition essentially requires that the intersection $V \cap L$ is empty for "most" such subspaces. This holds for the circle, as most points in the 2D plane do not lie on the circle itself.

The parameter $\beta$ is crucial because it bounds how many disconnected pieces of the variety we must cover when we intersect it with a subspace L. As our covering construction in the paper (following [41]) places the centers of ϵ-balls on these intersections $V \cap L$, $\beta$ directly influences the final covering number bounds by controlling the complexity of these intersections.

[41] Yifan Zhang and Joe Kileel. Covering number of real algebraic varieties and beyond: Improved453bounds and applications. arXiv e-prints, pages arXiv–2311, 2023.454.

Thank you very much for your careful review of our submission. We highly value your constructive feedback and suggestions.

Q1. How are $(\psi\_k, \mu\_k)$ chosen for later 2 cases?

The explicit definitions for the trial functions and measures used in these experiments are provided in Appendix G.2, as mentioned on line 330. These are written right after line 804 and right after line 818.

Q2. For the 1st case only Dirac measures are used, does the theory still hold for general (\mu_k)? If Dirac measures are used, justify why this is a good choice?

We thank the reviewer for this insightful question. The reviewer is correct that our original experiments focused heavily on the Dirac measure. To demonstrate that our theory holds for general measures, we have conducted an additional experiment for the Harmonic Oscillator using sine and cosine test functions (i.e., $\mathcal{T}$ includes basis functions from $\mathcal{B}$) with the Borel measure.

In this case, we obtain $d\_V=2$, just as with the Dirac measure. Consequently, the results are nearly identical to those originally shown in Figure 1 (a). The new results are as follows:

These results confirm that our core findings are robust to the choice of measure. We will add this result as a new figure in the Appendix.

It is important to note that the goal of our paper is not to prescribe the optimal choice of trial functions or measures, but rather to demonstrate that our theoretical framework holds across a variety of settings for $\mathscr{D}$, $\mathcal{B}$, and $\mathcal{T}$.

Q3. Revise the abstract/introduction to clearly clarify that D is assumed fully known. If the method cannot handle missing terms in D, state this limitation explicitly in future works.

We thank the reviewer for bringing this point to our attention. Our use of "incomplete knowledge" refers specifically to the source terms or boundary conditions, not the differential operator $\mathscr{D}$ itself, which we assume to be known, as detailed in Appendix C.

To resolve this confusion, we will revise the manuscript as follows:

• Line 26: prior knowledge of the governing differential equations, particularly their source terms or boundary conditions, is often incomplete.
• Line 34, 90: formulate the incomplete differential equation constraints
• Line 96: the discrete weak form of the known differential equation $\mathscr{D}$ is defined by

Q4: Can the framework handle unknown terms in D? If no, any thoughts?

Yes, our framework will be readily extended to a setting where the operator $\mathscr{D}$ has unknown parameters (e.g., an unknown diffusion coefficient $c$) that are learned simultaneously from the data. For instance, in the diffusion equation example, if $c$ were also to be learned, one could simply analyze the affine variety $\mathcal{V} = \{(w, c) : \langle \mathscr{D}_c[w^\top \phi], \psi_k \rangle = 0\}$ defined by polynomials over an augmented parameter space including both the solution weights w and the operator parameters c. We will add this point to the Appendix as a direction for future work.

However, to improve the main paper's clarity and focus, we will remove the ambiguous "incomplete" description regarding $\mathscr{D}$ and explicitly state that the operator is assumed to be known, as detailed in our response to Q3.

Q5: Through reading and many terms like "data-driven", it's not clear if the basis is fixed or learned.

The basis functions are fixed, not learned, as part of the modeling process. To clarify this, we will revise line 104 as follows:

Line 104: "Let B be a fixed basis (i.e., not learned like a neural network)."

Q6: I am also curious how does $d_V$ (and the final performance) change if the basis $\phi_j$ is misspecified (e.g., missing key frequencies)? You mentioned mis-used basis can downgrade performance, maybe show an ablation study?

This is an excellent question. We have performed the requested ablation study for the Harmonic Oscillator. In this experiment, the basis $\\{\phi\_j\\}$ is intentionally misspecified by omitting the known frequency of the analytical solution. We used the Dirac measure and a constant trial function, with $K := |\mathcal{T}|$ set sufficiently large. The results are below:

The results show that performance is poor and does not improve with more data $n$ and less dimension $d$. This is because a misspecified basis introduces a large approximation error (the inability of the hypothesis space to represent the true solution), which becomes the dominant factor over the estimation error that our theory bounds.

The decomposition of the total error is defined in Appendix C.4, Eq. (14):

where $f^*$ is the true function and $f^*\_{\\mathcal{H}}$ is the best possible approximation within the hypothesis $\\mathcal{H} = \\{ w^{\\top} \\phi : w \\in \\mathcal{V}\_R \\}$.

In this misspecified case, the approximation error term is large and irreducible. The physically constrained solution space may not contain any non-trivial solution, leading to $d_V=0$. We will add this experiment and discussion to the Appendix.

I've completed the experimental results and the detail draft, and I'd like to share them with you.

The experimental results for the diffusion equation, which correspond to fig:diff-perf-weak in main texts, are shown below. For details on the experimental setup and other information, please refer to the manuscripts of Section 5.2. The experimental results shown in fig:ho-perf-weak correspond to the weak solution of the harmonic oscillator. I have already provided these results in my rebuttal response to Q2.

Results for weak solution of diffusion equation

Manuscripts of Section 5.2 Learning Weak Solutions

\subsection{Learning Weak Solutions}

In this section, we investigate weak solutions for the harmonic oscillator and the diffusion equation, by employing the Dirac measure, which corresponds to the collocation method used in PINNs. The governing equations and basis functions are identical to those in Section 5.1.

\paragraph{Harmonic Oscillator}

We define the trial functions $\psi\_k$ for $1 \leq k \leq K\_t$ as:

where $\omega_k \coloneqq \frac{k\pi}{T}$ is the frequency. The associated measure $\mu_k$ is the Lebesgue measure on $\Omega = [0, T]$.

The dimension of the affine variety remains $d_{\mathcal{V}} = 2$, consistent with the strong solutions. Experimental results in \cref{fig:ho-perf-weak} confirm this. The performance of PILR is stable and independent of the number of basis functions, unlike RR, which shows performance degradation as model complexity increases.

\paragraph{Diffusion Equation}

The trial functions $\psi_{k, k'}$ combine a piecewise constant basis in time and a Fourier basis in space. For indices $1 \leq k \leq K_t$ and $1 \leq k' \leq K_x$, they are:

where $1_{[t_k, t_{k+1}]}(t)$ is the indicator function for the time interval, defined as

and $\omega_{k'} = \frac{k'\pi}{\Xi}$. The associated measure $\mu_k$ is the Lebesgue measure on $[-\Xi, \Xi] \times [0, T]$.

The dimension of the affine variety, $d_{\mathcal{V}} = 2 \min(d_x, d_t) + 1$, is identical to the strong solution case. The results in \cref{fig:diff-perf-weak} show that PILR's generalization performance remains robust as the number of spatial basis functions $d_x$ increases, demonstrating its advantage over RR.

Thank you very much for taking the time to review our manuscript. Your feedback is greatly appreciated.

Q1: The paper's assumptions are reasonable for theory but may not hold in practice.

We agree that acknowledging the potential gap between our theoretical assumptions and real-world application is crucial. We will add the following statement to Section 3 to clarify this for readers:

"While we assume Gaussian noise for analytical tractability, noise in observational data from physical phenomena can be far more complex. Bridging this gap is an important direction for future work."

Q2: The claim of 'incomplete knowledge' is confusing, as the operator $\mathscr{D}$ seems to be fully known.

We thank the reviewer for highlighting this ambiguity. Our use of "incomplete knowledge" refers specifically to the source terms or boundary conditions, not the differential operator $\mathscr{D}$ itself, which we assume to be known, as detailed in Appendix C.

To resolve this confusion, we will revise the manuscript as follows:

• Line 26: prior knowledge of the governing differential equations, particularly their source terms or boundary conditions, is often incomplete.
• Line 34, 90: formulate the incomplete differential equation constraints
• Line 96: the discrete weak form of the known differential equation $\mathscr{D}$ is defined by

Furthermore, we will add a note in the Appendix that our framework can be extended to cases where $\mathscr{D}$ is partially unknown (e.g., an unknown diffusion coefficient $c$). This would involve analyzing an affine variety $\mathcal{V}$ defined by polynomial constraints on an augmented parameter space that includes both the model weights $w$ and the unknown physical parameter $c$. For the linear diffusion equation, this would correspond to the variety $\mathcal{V} = \\{(w, c) : \langle \mathscr{D}_c[w^\top \phi], \psi_k \rangle = 0\\}$. We will add a note on this possible extension in the Appendix.

Q3: Could the author provide an example where something else than a Borel or Dirac measure for $\mu_k$ would be a useful choice?

Yes. A key example arises when extending the problem to differential equations on a curved manifold, such as the unit sphere $\mathbb{S}^2$. In such settings, it is natural to define the integral using the Riemannian volume measure $\mathrm{d}\mathrm{Vol}_{\mathcal{M}}$. For instance, Zhou and Shi (2025) recently adopted this approach on manifolds.

Zhou, Hanfei, and Lei Shi. "Weak Physics Informed Neural Networks for Geometry Compatible Hyperbolic Conservation Laws on Manifolds." arXiv preprint arXiv:2505.19036 (2025).

Q4: I'm still struggling to understand what are path-connected components in a $( \beta, d\_V )$-regular set. Could the author provide some intuition on this?

We appreciate the opportunity to provide more intuition. We have a detailed explanation in Appendix B.2, using the minimal example of a circle in a 2D plane, which we summarize here.

Consider a circle defined by $V = \\{ (x, y) \in \mathbb{R}^2 \mid x^2 + y^2 - 1 = 0 \\}$. The ambient space dimension is $d = 2$, and the circle's intrinsic dimension is $d_V = 1$. In this 2D plane, we can consider affine subspaces $L$, such as lines (1-dimensional) and points (0-dimensional). The definition of a $(\beta, d_V)$-regular set depends on the codimension of these subspaces, which is $\text{codim}(L) = d - \dim(L)$.

Case 1: Intersection with Lines

For any line $L$ in the plane, its dimension is $\dim(L) = 1$. The codimension is therefore $\text{codim}(L) = 2 - 1 = 1$. Since $\text{codim}(L) \leq d_V$ (as $1 \leq 1$), this falls under the first case of the definition. Here, $\beta$ is defined as the maximum number of path-connected components of the intersection $V \cap L$.

Let's see what happens when we intersect our circle $V$ with a line $L$:

• No intersection: If the line misses the circle, $V \cap L$ is empty (0 components).
• Tangent line: If the line touches the circle at a single point, $V \cap L$ is a single point (1 path-connected component).
• Secant line: If the line cuts through the circle, $V \cap L$ consists of two distinct points. Since these points are separated, this set has two path-connected components.

The maximum number of path-connected components we can get is 2. Therefore, for the circle, $\beta = 2$. Intuitively, $\beta$ is related to the topology of the variety $V$. A line can intersect the single "loop" of a circle at most twice.

Case 2: Intersection with Points

Now, consider a point $L$ in the plane. Its dimension is $\dim(L) = 0$. The codimension is therefore $\text{codim}(L) = 2 - 0 = 2$.

Since $\text{codim}(L) > d_V$ (as $2 > 1$), this falls under the second condition, which requires $V \cap L$ to be empty for "most" such subspaces. This condition essentially requires that the intersection $V \cap L$ is empty for "most" such subspaces. This holds for the circle, as most points in the 2D plane do not lie on the circle itself.

The parameter $\beta$ is crucial because it bounds how many disconnected pieces of the variety we must cover when we intersect it with a subspace L. As our covering construction in the paper (following [41]) places the centers of ϵ-balls on these intersections $V \cap L$, $\beta$ directly influences the final covering number bounds by controlling the complexity of these intersections.

[41] Yifan Zhang and Joe Kileel. Covering number of real algebraic varieties and beyond: Improved453bounds and applications. arXiv e-prints, pages arXiv–2311, 2023.454.

I want to thank the authors for engaging constructively in the review process, and answering my questions with an educative example. I appreciated the paper, which I believe will make a significant contribution to the field, and will be happy to see it published. I have no further comments.