Understanding Generalization in Physics Informed Models through Affine Variety Dimensions

Q1. This paper only takes linear regressors on at most two input dimensions.

A1. Our input setting is not limited to two dimensions. As noted in lines 151–168, our analysis handles general $m$-dimensional input, i.e., $x\_i \in \Omega \subset \mathbb{R}^m$.

---

Q2. What does the covering number of balls in hypothesis space mean?

A2. The covering number measures the capacity of a hypothesis space. As this is a standard concept found in basic machine learning textbooks such as [1], we assume readers are familiar with it.

[1] https://cs.nyu.edu/~mohri/mlbook/

---

Q3. Clarify what "physics-informed" means.

A3. As stated on line 22, "physics-informed" refers to models incorporating the structure of differential equations. Specifically, the operator $\mathscr{D}$ is used to constrain the model via $\mathcal{T}$ and $\mathcal{B}$.

---

Q4. Do the results assume:
(1) all basis functions $\phi$ are known,
(2) the hypothesis space is separable,
(3) it can represent the target function?

A4. (1) and (3) are correct, but (2) is not. Due to noise, empirical loss may not reach zero.
We will revise Sec. 3.2 (around line 149) to state:

• All basis functions $\phi$ are known.
• The hypothesis space covers the target function, i.e. $f^* = \beta^* \phi$.

---

Q5. Many of the used symbols are unclear.

A5. If any parts remain unclear, we would appreciate a more specific explanation of what is confusing.

• $\mu\_k$ is a measure, and $\mathcal{T}$ is a set of test function-measure pairs (lines 119–126).
• $\phi = (\phi\_1, \dots, \phi\_d): \mathbb{R}^m \rightarrow \mathbb{R}^d$ will be defined explicitly around line 150. $\Phi$, $\phi_j$ are defined in lines 149–150 and 166.
• $\lambda$ is the eigenvalue of $C M^{-1} C$ (defined in Eq. 10).
• $x\_i \in \Omega$ denotes a sample, as in lines 154–155. $x$ was a generic element in $\Omega$, but we removed it as unnecessary.
• $m$ is the input dimension, and $d$ is the number of basis functions/parameters, as evident from the definition of $f^*$, $\mathcal{H}$ and $\beta$.
• “Path-connected” is a standard concept in math defined as follows. "$X$ is not path-connected if there exist disjoint open sets $A, B \subset \mathbb{R}^d$ such that $X \subset A \cup B$ and $A \cap X \ne \emptyset$, $B \cap X \ne \emptyset$."

---

Q6. Clarify the sentence: “The generalization capacity is determined by the local size of the hypothesis space ..."

A6. Recent work [2][3] shows generalization bounds can depend on data and algorithm choice. These factors effectively restrict the hypothesis space to a smaller, data-dependent subset, enabling tighter bounds. Our sentence highlights this idea and suggests that these aspects offer opportunities for deeper analysis of our bound.

[2] Bartlett, Peter L., et al. "Local Rademacher complexities."
[3] Steinke, Thomas, et al. "Reasoning about generalization via conditional mutual information."

---

Q7. Clearly state the assumptions in an explicit form.

A7. Due to space limits, Assumptions 1–3 were placed in Appendix B. We will move them to the main text between Lem. 3.1 and Thm. 3.2.

---

Q8. the theoretical results do not bring stronger conclusions and estimating $d\_V$ is not easy.

A8. Our results provide a stronger theoretical contribution by revealing how the structure of general (including nonlinear) differential equations affects generalization through affine varieties—an aspect not addressed in prior work. Moreover, the estimation of $d\_V$ is not difficult as described in Sec. 4.2.

---

Q9. First, it is not necessary to provide experiments on model performance. Second, the most important thing is to verify the theoretical conclusions.

A9. Our experiments are designed to support our primary theoretical claim: a smaller affine variety dimension $d_V$ leads to better generalization. If there are other experimental setups you consider more appropriate, we would be happy to hear your suggestions.

---

Q10. Comparing PILR with RR is unfair; kernel methods should be used.

A10. We believe the comparison is valid. RR uses feature map $\phi$ and constrains $\beta$ in $\mathbb{B}\_2(R)$, with estimator:

$$\hat{\beta} = (\Phi^{\top}\Phi + n I)^{-1} \Phi y$$

PILR modifies the constraint to $\mathcal{V}\_R$ via differential equations. Comparing RR on $\mathbb{B}\_2(R)$ and PILR on $\mathcal{V}\_R$ is natural. Kernel ridge regression uses RKHS norms, which are not directly comparable.

Q1. The bound given by Theorem 3.2 is trivial and can be derived without the analyzing tools used. Could you give a tighter bound on the minimax risk?

Our generalization bound cannot be derived without the analyzing tools we introduced, particularly the covering number of the affine variety. Our bound is sufficiently tight to explain the performance gap between the physics-informed and non-physics-informed linear regressors.

While it is true that tighter bounds may be obtained in future work, e.g., by considering algorithm-dependent generalization bounds [1], such bounds mix the effects of the hypothesis space and the learning algorithm. This makes it difficult to isolate and analyze the contribution of the physical structure. Therefore, while tightness is desirable, it does not necessarily lead to better understanding of the underlying mechanisms.

[1] Steinke, Thomas, and Lydia Zakynthinou. "Reasoning about generalization via conditional mutual information." Conference on Learning Theory. PMLR, 2020. (https://proceedings.mlr.press/v125/steinke20a.html)

Q2. Could you take a step further to identify which property PINN possesses enables good generalizing capacity?

As stated in the conclusion, our analysis is limited to linear models, and extending it to neural networks is left as future work. On the practical side, our contribution lies in determining the appropriate size of $|\mathcal{T}|$ (the number of PDE evaluation points in the PINN setting) via $d_V$ for each equations, which is a nontrivial insight. Moreover, reliability guarantees in numerical analysis are valuable in engineering applications in their own right. These contributions are expected to carry over when extended to more general neural networks.

Q1. Assumption 2 requires positive min. singular value of X, which typically holds only if n ≥ d...

A1. Since $\hat{\beta} - \beta^* \in \mathbb{B}_2(2R)$, Eq. (12) still holds under a weaker condition. For $\kappa > 0$,

$$\frac{1}{\sqrt{n}}\\|\Phi\beta\\| \geq \sqrt{\kappa} \\|\beta\\|_2,\quad \forall \beta \in \mathbb{B}_2(2R).$$

If $R$ is sufficiently small, it is reasonably expected that $\beta$ will not lie in the eigenspace corresponding to the eigenvalue 0 of $\Phi$. We will reflect this relaxation in the manuscript.

---

Q2. The key lemma is from Zhang & Kileel (2023); Thm 3.2 seems like a direct application...

A2. While we rely on the covering number bound from Zhang & Kileel (2023), applying this result to the analysis of physics-informed models is itself a nontrivial contribution. Our contribution lies in formulating the generalization problem from the viewpoint of affine varieties—a perspective not previously explored in the ML or PIML literature.

---

Q3. Affine variety dim. vs. generalization resembles rank-based arguments...

A3. This is an insightful point. However, the design matrix $\Phi$ in our setting is not inherently low-rank. The low-rank property emerges by imposing the structure of the differential equations. One of the main contributions can be viewed as reducing the behavior of physics-informed models to rank-based arguments—a nontrivial step that helps uncover their underlying structure.

In addition, rank-based analysis is limited to linear differential equations. Our use of the affine variety dimension generalizes this to nonlinear cases, offering a more broadly applicable perspective in ML and physics-informed contexts.

---

Q4. Does the gen. bound depend on $|\mathcal{T}|$? If so, how...

A4. Yes. $|\mathcal{T}|$ determines the number of constraints. As $|\mathcal{T}|$ increases, the variety $\mathcal{V}$ becomes smaller:

$$\mathcal{T}\_1 \subset \mathcal{T}\_2 \Rightarrow \mathcal{V}(\mathscr{D},\mathcal{B},\mathcal{T}\_2) \subset \mathcal{V}(\mathscr{D},\mathcal{B},\mathcal{T}\_1) \Rightarrow d\_{\mathcal{V}(\mathscr{D},\mathcal{B},\mathcal{T}\_2)} \leq d\_{\mathcal{V}(\mathscr{D},\mathcal{B},\mathcal{T}\_1)}.$$

We will add this clarification around lines 168–178.

---

Q5. Clarify why Thm 3.3 upper bound shrinks as affine dim. decreases...

A5. Since $D^\top G D$ is positive semi-definite ($\lambda_j \geq 0$), we have $1/(1+\xi) > 1/(1+\xi\lambda_j)$. As $d_V$ decreases, more terms in the bound use $1/(1+\xi\lambda_j)$ instead of $1/(1+\xi)$, leading to a smaller total. We will clarify this and emphasize the semi-definite property of $G$ in Definition 3.1 and Theorem 3.3.

---

Q6. In Fig. 2, test loss increases with parameters in some cases... Contradiction?

A6. There is no contradiction. Our theory provides an upper bound, and fluctuations in performance within that bound are expected. Additionally, the test loss may be influenced by the choice of the hyperparameter $\lambda$. Furthermore, our bound includes a $\log d$ term in the first component, so it is not entirely independent of $d$.

---

Q7. Can you design experiments varying $d_V$ while fixing $d$?

A7. Yes. By varying $|\mathcal{T}|$, we control $d_V$ independently of $d$, and we provide results based on this setting.

Linear Bernoulli

Nonlinear Bernoulli

Linear Heat

Nonlinear Heat

Observation:
Across all settings, test MSE increases as the dimension $d_{\mathcal{V}}$ grows.