Uncertainty Quantification for Physics-Informed Neural Networks with Extended Fiducial Inference

[W1.] Do we still require a prior distribution in EFI?

In Appendix~A1, we reviewed the original EFI theory, which requires imposing a sparse prior on $w$ to guarantee posterior consistency for the hypernetwork. Instead of adopting a Bayesian approach on the hypernetwork, we designed a narrow-neck network architecture to eliminate the need for prior assumptions on $w$.

[W2.] Why is $\delta_g = 0$?

Thanks for pointing this out. As mentioned in Theorem 4.2 of Liang et al. (2025), since we used the full dataset to calculate the gradient, the stochastic gradient reduces to the true gradient. Hence, $\delta_g = 0$.

[1] Liang, F., Kim, S., and Sun, Y. Exended fiducial inference: Toward an automated process of statistical inference. Journal of the Royal Statistical Society Series B, 87:98–131, 2025.

[W3.] Eigenvalue assumption

In real-world implementations, when the penalty coefficient $\eta_\theta$ in Equation~(10) is large, the estimated parameters $\hat{\theta}\_i$ tend to be almost identical across all $i$. Since the $\hat{\theta}\_i$'s are highly correlated, the same is true for the corresponding $m_i$'s. As a result, $\mathbf{E}[m_i m_i^T]$ is nearly a rank-1 matrix, with one dominant eigenvalue and all others close to zero. On the other hand, if $\eta\_\theta$ is not sufficiently large, more variation is allowed across the $\hat{\theta}\_i$'s, and the lower bound for the minimum eigenvalue of $\mathbf{E}[m_i m_i^T]$ will be slightly larger. In theory, as $\eta_\theta \to \infty$, the bottleneck size could be reduced to $1$. In practice, however, $\eta_\theta$ does not approach infinity, so we choose a slightly larger bottleneck size to promote more robust convergence of the algorithm while still satisfying the eigenvalue assumption.

[W4.] Interpretation of Theorem 3.2

The asymptotic consistency of the inverse function essentially ensures a one-to-one correspondence between $Z_n$ and $\theta$; that is, different realizations of the random error $Z_n$ lead to different estimates of $\theta$, thereby ensuring that the uncertainty in $Z_n$ is properly propagated to $\theta$ and the uncertainty in $\theta$ is accurately quantified.

To be honest, I am not able to verify the correctness of the authors' response. This may be due to the simplicity or lack of clarity in the author's rebuttal provided, or it may reflect my own limited familiarity with the relevant literature. Given that the paper builds heavily on Liang et al. (2025), which has only recently been accepted to JRSSB, it is understandably difficult for most readers to properly assess the current paper.

1. It seems that a sparse prior distribution is still needed for EFI framework. Even after reading the paper again, I still do not understand why a narrow-neck neural network eliminates the need for a prior.

2. it relies heavily on Liang et al. (2025), which I am not able to verify its correctness.

3. I can understand the first half of the response. However, for the second half, why the lower bound for the minimum eigenvalue of $\mathbf{E}[m_i m_i^T]$ increases, when more variation is allowed across the $\hat{\theta}_i$?

4. I do not understand author's response. I am asking about the non-asymptotic convergence of the inverse function with finite samples; while the author's response tells me how the uncertainty in $Z_n$ is properly propagated to $\theta$. I am completely lost here.

Thank you for your prompt and valuable comments. Your advice will be incorporated into our revision.

[Q1]

We apologize for any confusion. In the original EFI framework by Liang et al. (2025), a sparse prior is imposed on the $w$-network to achieve consistency for the inverse function estimator. As stated on page 2, one of the goals of the present paper is to develop a new EFI framework by employing a narrow-neck $w$-network, such that the sparse prior on the $w$-network can be eliminated, while still achieving consistency for the inverse function estimator. We establish this consistency in Theorem 3.2.

In the proof, we introduce an auxiliary stochastic neural network (StoNet), which is constructed by randomizing the feeding values of the hidden units at the neck-layer by adding a small random error (see Equation (12) of the present paper). This StoNet was originally developed in Liang et al. (2022) and Sun & Liang (2022), and it
has been shown to be able to bridge statistical theory from linear models to deep neural networks (Sun & Liang, 2025). Specifically, StoNet is asymptotically equivalent to the conventional DNN as the size (standard deviation) of the added random errors approaches 0 (see Liang et al. (2022)} for the proof). Building on the properties of the StoNet, we show in the present paper
that the narrow-neck structure ensures the inverse function can be consistently estimated.

This result is extremely important for EFI, as it completely eliminates the need for priors, aligning with the core goal of fiducial inference --- inferring the uncertainty of model parameters based solely on observations.

[1] Liang, F., Kim, S., and Sun, Y. Exended fiducial inference: Toward an automated process of statistical inference. Journal of the Royal Statistical Society Series B, 87:98–131, 2025.
[2] Liang, S., Sun, Y., and Liang, F. Nonlinear sufficient dimension reduction with a stochastic neural network. NeurIPS 2022, 2022.
[3] Sun, Y. and Liang, F. A kernel-expanded stochastic neural network. Journal of the Royal Statistical Society Series B, 84(2):547–578, 2022.
[4] Sun, Y. and Liang, F. Uncertainty quantification for large-scale deep neural networks via post-stonet modeling. Statistica Sinica, pp. in press, 2025. URL https://www3.stat.sinica.edu.tw/ss_newpaper/ SS-2024-0294_na.pdf.

[Q2]

The concept of EFI presented in the paper is from Liang et al. (2025) and a more recent work Kim and Liang (2025); both work involves solid theoretical studies. However, as mentioned above, the present work represents an important development of EFI. It completely eliminates the need for priors, aligning with the core goal of fiducial inference — inferring the uncertainty of model parameters based solely on observations.

[5] Kim, S. and Liang, F. Extended fiducial inference for individual treatment effects via deep neural networks. Statistics and Computing, 35, 2025. URL https://api.semanticscholar.org/CorpusID:278326914.

[Q3]

Thank you for the thoughtful question. In a nutshell, the output of the $w$-network and the output of the last hidden layer together forms a multivariate regression (see equation (11) of the paper) under the framework of the stochastic neural network: $\hat{\theta}_i=\Xi^T m_i +e_i, \quad i=1,2,\ldots,n$ where $\hat{\theta}_i=(\hat{\theta}_i^{(1)}, \ldots, \hat{\theta}_i^{(p)}))^T \in \mathbb{R}^{p}$, $m_i \in \mathbb{R}^{d_h}$, $\Xi=(\xi_1,\ldots,\xi_p) \in \mathbb{R}^{d_h\times p}$, and $e_i \in \mathbb{R}^p$. However, this regression differs from conventional regression, where $\hat{\theta}_i$ can be treated as known (as $\epsilon \to 0$), while $m_i$ is random (imputed conditioned on the input and output of the network). In the limit, as the standard deviation of $e_i$ approaches 0, we have $(\hat{\theta}_1, \ldots,\hat{\theta}_n)^T=(m_1,\ldots,m_n)^T \Xi,$ or, in matrix form, $\hat{\Theta}= M \Xi,$ Therefore, $rank(\hat{\Theta}) \leq rank(M)$.

In our experiments, we observe that if $\eta_{\theta}$ is small, which increases the variation of $\hat{\Theta}$, the smallest eigenvalues of $M$ increases. In fact, what we intend to convey is that if $\eta_{\theta}$ is small, we need to adjust the neck-layer to be slightly wider. By appropriately choosing $\eta_{\theta}$ and the neck-layer width $d_h$, the eigenvalue lower bound condition can be satisfied.

[Q4]

We apologize for the misunderstanding — we initially thought you were requesting an explanation of the ``practical implications of this result.''

Regarding the convergence rate of the inverse function estimator (i.e., non-asymptotic convergence with finite samples), we note that this has been implicitly established in our proof of Theorem 3.2. As indicated by equations (A20) and (A21) (in the supplementary material), the convergence rate is $O(1/\sqrt{n})$, which is an ideal rate.

[W1.] Carried over from EFI: Compared to variational inferences, the uncertainty modeling approach has few free hyper-parameters and does not need explicit prior modeling. However, some typical drawback of MCMC carry over: (algorithm 1: burn-in periods, difficulty to check convergence, sampling process usually slower than energy-loss minimization. perhaps wall-clock comparison of the algorithm compared to dropout, ensembling and the likes?)

In Table 1, we report the wall time for the Poisson-1D experiment using different algorithms. As shown, EFI is slower than PINN (with dropout) but much faster than B-PINN with the HMC sampler. Importantly, we have demonstrated that EFI is the only existing method capable of correctly constructing confidence intervals in a statistically rigorous manner. Since we employ a narrow-neck hypernetwork structure, the total number of parameters depends linearly on the size of the neck layer. Therefore, EFI is scalable with respect to the size of the $\theta$-network. For even larger networks, we can adopt transfer learning techniques by constructing EFI hypernetworks only for the last few layers of the $\theta$-network, which would significantly reduce the computational cost.

It is worth noting that the EFI training process is based on the Adaptive SGMCMC algorithm, which alternates between one step of SGD and one step of SGLD. Since both SGD and SGLD are stochastic gradient-based algorithms, EFI has the same order of sample complexity as standard optimization-based approaches. However, because EFI aims to construct accurate confidence intervals, it generally requires more training time compared to purely optimization-focused methods.

Table: Wall time for PINN, B-PINN and EFI-PINN
(Device: MacBook Pro M4 Pro CPU)

[W2.] It is unclear whether epistemic uncertainty is truly captured by the approach (or if it's both epistemic and aleatoric which can't be disentangled, see questions below). There is a lack of discussion on the epistemic and aleatoric uncertainty concept in Section 2. Also the approach appears to share similarity with the BNN+LV approach (see e.g. Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning). In Appendix A5 these concepts are discussed, but very limited $y = f(x) + \epsilon$ (normal, uniform noise) is quite limited noise assumption. Will the method still work if the noise is heteroscedastic? what about the most general case, where if $y = f(x,z)$ and $z$ is from some distribution? Can you refute the following statement: "EFI conflates epistemic & aleatoric uncertainty"?

As explained below (see our replies to Questions), EFI integrates epistemic & aleatoric uncertainty in a mathematically rigorous way. The method still works if the noise is heteroscedastic. In this case, we can set $\epsilon$ to a very small value, enabling the noise to be recovered. Actually, this has been tested in Liang et al. (2025), where the example in Section 5.4 demonstrates that EFI works with heteroscedatic noise (or outliers).

Yes, the method also works in the most general case, where $y=f(x,z)$. Refer to Liang et al. (2025) for the theoretical development on this case, where the random error does not appear in the model in an additive form.

[1] Liang, F., Kim, S., and Sun, Y. Exended fiducial inference: Toward an automated process of statistical inference. Journal of the Royal Statistical Society Series B, 87:98–131, 2025.

[Questions]

Thank you very much for the thoughtful comments. EFI aims to learn an inverse mapping for the model parameters, i.e., $\theta=G(Y_n,X_n,Z_n)$, through which the uncertainty of $\theta$ can be derived from the uncertainty of $Z_n$ (in the form of variable transformation as shown in equation (8) of the paper).

To make equation (8) (of the paper) more interpretable in the context of uncertainty quantification, we can rewrite it in a less formal manner as follows:

where $\pi(Z_n|Y_n,X_n)$ provides a measure for the aleatoric uncertainty, while $\pi(\theta|Y_n,X_n,Z_n)$ provides a measure for the epistemic uncertainty (whose variation can decrease to 0 as $n \to \infty$) and depends on the aleatoric uncertainty. That is, the uncertainty measured by EFI has integrated both types of uncertainty. We agree with Kirchhof et al. (2025) that distinguishing between these two types of uncertainty is not always feasible and one can depend on the other.

To better explain this concept, let's consider a linear regression example:

where $Z$ follows the standard Gaussian distribution. As shown in Liang et al. (2025), an
inverse function for $\beta$ is given by

Integrating with respect to the fiducial density of $Z_n$, the fiducial density of $\beta$ is given by

This theoretical result has been numerically verified in Liang et al. (2025) through the implementation of EFI.

Next, let's consider the binary response example you provided. Given a set of $Z$-values, directly solving the data-generating equation leads to

for $i=1,2,\ldots,n$.

Let $(z_i^{(1)}: i=1,2,\ldots,n)$, $\ldots$, $(z_i^{(m)}: i=1,2,\ldots,n)$ denote $m$ imputations from $\pi(Z_n|Y_n,X_n)$ in EFI, which measures the aleatoric uncertainty. By empirically integrating out $Z_n$ as what does in EFI, we obtain

where $\hat{p}$ is given by \hat{p}= \frac{1}{m} \sum_{k=1}^m \frac{ \\# \\{i: z_i^{(k)}=1\\} }{n}, whose variance decreases to 0 as $n\to \infty$. Again, the uncertainty of $\theta$ has integrated both the aleatoric and epistemic uncertainty. EFI can be implemented for the example by replacing its latent variable sampling algorithm with a Metropolis-based algorithm (dealing with discrete values of Z).

Thank you very much for your encouraging comments.

Error heterogeneity: We refer to Section 5.4 of Liang et al. (2025) and Figure 5 therein. Although the example is primarily designed for outlier detection, it also serves as an effective illustration of parameter estimation under heterogeneous error structures. The results demonstrate that the proposed method is robust to heterogeneous errors.

Computational efficiency: We acknowledge that the method is slower than variational inference (VI), primarily due to its sampling-based nature, as the method draws samples of $\theta$ from its fiducial distribution. However, the trade-off is our method's ability for uncertainty quantification. Moreover, our method is still scalable with respect to big data problems.

Self-diagnostic capability: The method possesses a built-in self-diagnostic mechanism, as it must converge to a zero-energy solution to ensure valid statistical inference. This feature enhances usability and robustness, particularly in accommodating variations in hyper-network parameters, which are typically non-identifiable following the general property of deep neural networks. The zero-energy convergence feature also partially address the issue of multi-modality (that you raised), which ensures the convergence to a solution to the data-generating equation.

Conceptual contribution and practical potential: We agree with your assessment that the paper offers a novel conceptual framework for uncertainty quantification. We also believe that the proposed method has strong practical potential. Notably, it can be extended to large-scale models, such as ResNets and CNNs, via transfer learning, as discussed at the end of the paper. Compared to existing approaches, the proposed method not only applies to a wide range of problems but also provides a rigorous framework for uncertainty quantification.

[1] Liang, F., Kim, S., and Sun, Y. Exended fiducial inference: Toward an automated process of statistical inference. Journal of the Royal Statistical Society Series B, 87:98–131, 2025.

[W1.] The paper lacks a thorough discussion of related work in general uncertainty quantification, particularly outside the context of PINNs.

In recent years, a variety of methods have been developed for uncertainty quantification in machine learning models, such as conformal prediction (Vovk et al., 2005), deep ensembles (Lakshminarayanan et al., 2016), stochastic deep learning (Sun & Liang, 2022; Liang et al., 2022). However, these methods primarily focus on prediction uncertainty and are either ineffective or inapplicable for quantifying uncertainty in model parameters. In contrast, EFI is applicable to both prediction uncertainty and model parameter uncertainty, and it further provides theoretical guarantees for the validity of the resulting prediction and confidence intervals.

[1] Vovk, V., Gammerman, A., and Shafer, G. Algorithmic Learning in a Random World. Springer, 2005.
[2] Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems, 2016. URL https://api. semanticscholar.org/CorpusID:6294674.
[3] Sun, Y. and Liang, F. A kernel-expanded stochastic neural network. Journal of the Royal Statistical Society Series B, 84(2):547–578, 2022.
[4] Liang, S., Sun, Y., and Liang, F. Nonlinear sufficient dimension reduction with a stochastic neural network. NeurIPS 2022, 2022.

[W2.] The evaluation of the proposed UQ method is limited—there is no direct assessment of uncertainty quality.

Below we provide an example, where the length of intervals provides a direct assessment of uncertainty quality. See Q2.

[W3.] There is a lack of real-world experiments to demonstrate the practical utility and robustness of the proposed approach.

We are not sure what you are asking for. In the paper, we have provided some real data examples in the context of PINNs. Regarding real-world applications of the original EFI method, we refer to Kim & Liang (2025), where EFI was applied to infer the uncertainty of individual treatment effects (with real data examples). Notably, as shown in Kim & Liang (2025), EFI eliminates the need for the central limit theorem in uncertainty inference for complex models, as well as the requirement for specifying the exact form of the target model. Additionally, for potential applications of EFI on classification problems, see Q3.

[5] Kim, S. and Liang, F. Extended fiducial inference for individual treatment effects via deep neural networks. Statistics and Computing, 35, 2025. URL https://api.semanticscholar.org/CorpusID:278326914.

[Q1.] In line 91, a distribution for the random error is required—does that function similarly to a prior? I don’t think specifying a prior is a major drawback of Bayesian methods. In fact, incorporating informative priors, such as knowledge-based priors, can improve performance. Even if the true prior is unknown, using a Gaussian prior with small variance generally does not hurt performance.

A distributional assumption for the random error is essential for any likelihood-based approach; without it, the likelihood function cannot be constructed. In Bayesian methods, a prior is specified only for the model parameters. While a well-specified prior—particularly one based on domain knowledge—can enhance statistical analysis, it can also introduce challenges. In small-n-large-p settings, such as Bayesian deep neural networks, an improperly specified prior may overwhelm the information from the data, leading to biased inference. To mitigate this issue, a theoretical justification for posterior consistency is generally required.

In the context of Bayesian PINNS, the PDEs themselves act as a form of prior, constraining the neural network parameters to lie on a sub-manifold. For instance, consider the 1-D Poisson example in Section 4.1. In Bayesian PINNs, a Gaussian prior has to be introduced to constraint the network parameters 2 to satisfy the required PDE. However, the resulting uncertainty heavily depends on the variance $\sigma_f^2$ (see Table 1 in manuscript). This indicates the dependence of statistical inference (for the model) on the prior hyperparameters, which is undesirable for a scientific method. In contrast, EFI offers a mathematically rigorous framework for incorporating PDE constraints into neural network training, enabling accurate uncertainty quantification for both predictions and model parameters—without relying on prior distributions.

[Q2.] More discussion is needed around general uncertainty quantification methods. There are various approaches such as MCMC methods, variational inference, and ensemble-based methods. I couldn’t find a direct evaluation of uncertainty quality. Instead of relying only on MSE to assess regression accuracy, how can we compare the predicted uncertainty against ground-truth uncertainty?

See W1 for discussions on general uncertainty quantification methods. We appreciate the feedback regarding the evaluation of our uncertainty quantification (UQ) method. In the context of Physics-Informed Neural Networks (PINNs), the ground-truth uncertainty is inherently unknown, making direct comparison between predicted and true uncertainty challenging. To address this, we adopt a statistical approach by assessing the reliability of predicted uncertainties through confidence intervals and their coverage rates. This is a well-established practice in statistics when ground-truth uncertainty is unavailable.

To further validate our method, we conducted experiments on simpler problems (e.g., linear regression) where analytical uncertainty estimates (e.g., from Ordinary Least Squares, OLS) are available and serve as the ground truth. While applying our proposed EFI algorithm to such problems may seem excessive, it serves as a sanity check. By comparing EFI-derived confidence intervals with those from OLS, we demonstrate that EFI produces well-calibrtaed uncertainty estimates (nearly same as the ground truth). Below is an example of this verification:

Consider a simple linear regression model: $y_i = \beta_0 + \beta_1 x_i + \epsilon_i,\quad i=1,\dots, n$ where we set $\beta_0=-1$ and $\beta_1=2$. Linear regression can be reinterpreted within the framework of PINNs as follows: $y_i^u = u_{\theta}(x_i^u) + z_i^u, \quad i=1,2,\dots,n_u,$ $0 = \frac{d^2}{dx^2} u_{\theta}(x_i^f), \quad i=1,2,\dots,n_f.$

Here, $u_\theta$ is a DNN with no pre-imposed structure, while the assumption of linearity is enforced through the differential equation constraint. This formulation seamlessly integrates linear regression into the PINN framework. For EFI, we can derive $\beta_1$ estimates through automatic differentiation, where $\beta_1 = \frac{d}{dx}u_\theta(x)|_{x=0}$. In table below, we show that EFI obtained almost the same confidence interval as OLS.

Table: Confidence intervals for OLS and EFI

[Q3.] UQ is widely applied in computer vision and has been evaluated on various image classification tasks. The proposed method could also be tested on more challenging real-world tasks to better demonstrate its value. Can the method apply on classification tasks?

Although this paper focuses on UQ for PINNs, EFI framework can be extended to classification and logistic regression scenario. Consider a logistic regression: $P(y=1|x, \theta) = \frac{1}{1 + \exp\{-x^T\theta\}}$ We can adopt EFI framework by modifying the energy function as follows: $\sum_{i=1}^n \rho((z_i-x_i^T \theta)(2y_i-1)),$ where $\rho(\cdot)$ is the ReLU function, and $z_1,z_2,\ldots,z_n \stackrel{iid}{\sim} Logistic(0,1)$ with the CDF given by $F(z)=1/(1+e^{-z})$. In this experiment, the samples $x_i$ are generated from two 2D Gaussian distributions with means $\mu_1 = (2, 2)$ and $\mu_0 = (-2, -2)$, and a covariance matrix $\Sigma = 2.56 I_2$. Each class contains 500 samples. The true parameter is set to $\theta = (1, 1)$. We are able to achieve 95% coverage rate on the parameter $\theta$. We can also incorporate DNN structure to adapt nonlinear decision boundaries: $\sum_{i=1}^n \rho((z_i-h(x_i;\theta))(2y_i-1)),$ where $h(\cdot;\theta)$ is an arbitrary DNN parameterized by $\theta$. Furthermore, we can extend EFI framework to multiclass classification problem. We define:

Then the fitting error term for multi-class logistic regression can be defined as follows:

Thank you very much for your encouraging comments and for kindly raising the score to 4.

Confidence Interval. Also, thank you for your thoughtful question regarding the use of confidence intervals in evaluating the quality of uncertainty quantification (UQ). In our approach, the following two metrics are primarily considered:

— Coverage rate: For a nominal 95% confidence interval, approximately 95% of the true values should fall within the interval across test samples. This calibration metric directly reflects how well the uncertainty estimates align with their underlying true values.

— Interval width: It reflects the size of the uncertainty. Narrower intervals are desirable if they maintain correct coverage, as they indicate sharper confidence intervals — that is, the model’s ability to provide informative and precise rather than overly conservative uncertainty estimates.

In summary, confidence intervals serve as a diagnostic tool for assessing UQ quality by evaluating their coverage and width, which together characterize the reliability and informativeness of the model’s uncertainty estimates.

Practical Applications: We believe that the proposed method has strong practical potential. Notably, it can be extended to large-scale models, such as ResNets and CNNs (with applications to image data), via transfer learning, as discussed at the end of the paper.

Q1: Concrete Dropout.

Consider Concrete Dropout by Gal et al. (2017). Unlike the traditional dropout method using fixed dropout rate, Concrete Dropout treats the dropout rate as a hyperparameter and learns it simultaneously when training the model.

Table 1 Metrics for 1D-Poisson with Concrete Dropout

We applied Concrete Dropout with the code provided on https://github.com/yaringal/ConcreteDropout. Table 1 shows the result of 1-D Poisson model with Concrete Dropout. All necessary algorithm parameters in Concrete Dropout we applied were set as the code provided.

Compared to our proposed EFI algorithm, Concrete Dropout achieves a comparable MSE, indicating similar model estimation accuracy. However, it performs significantly worse in terms of uncertainty quantification, with a coverage rate of only 11.85%, far below the nominal 95%. This substantial under-coverage is due to an underestimation of predictive uncertainty, as evidenced by the markedly narrower confidence interval width.

From another perspective, given that the performance metrics of Concrete Dropout are relatively similar to those of the PINN algorithm without dropout, we suspect that it effectively learns a dropout rate that is quite close to zero.

More importantly, regardless of whether the learned dropout rate is small or large, the method lacks theoretical justification or principled guidance to ensure that the resulting predictive uncertainty aligns with the true underlying uncertainty. Furthermore, it is not applicable for quantifying uncertainty in model parameters. In contrast, EFI provides a unified framework for quantifying both predictive uncertainty and parameter uncertainty.

Q2: Other Bayesian methods

We also attempted to implement the method proposed in Implicit Weight Uncertainty in Neural Networks in the context of PINNs. However, to the best of our knowledge, there is no existing literature that reports such an implementation. Our attempt failed to converge to the desired PDE solution. Due to time constraints, we were unable to thoroughly tune the hyperparameters, although we have tried very hard. In contrast, the existing Bayesian approach, B-PINN, as demonstrated in our paper, has already shown that Bayesian methods fail to construct faithful confidence intervals for PINNs.

Thank you for adding the concrete dropout results. Upon further reflection, I have an additional question. For the trained Bayesian neural networks, was homoscedastic regression without a learned prediction variance, homoscedastic regression with a learned prediction variance, or heteroscedastic regression used? If homoscedastic regression without a learned prediction variance was used, as seems to be the case from Lines 287-297, then the confidence interval results for the Bayesian neural networks take into account only epistemic uncertainty and not aleatoric uncertainty, which is based on the prediction error. This makes me question the strength of the Bayesian neural network results.

Please correct me if I am wrong. EFI seeks to learn a distribution for theta based on the prediction errors. Looking at the variability of predictions for thetas sampled from an approximation of the posterior using a Bayesian method would give you information on how much a prediction for an input is constrained by the training data, as can be seen in Figure 3 of Yang et al. (2021). It may not inherently be linked to prediction error. However, aleatoric uncertainty is learned by taking prediction error into account. I understand that this is different from how the confidence interval would be constructed for EFI, but it is, to my knowledge, the way that a Bayesian method would model prediction error. I'd like to see how well that performs. I acknowledge the use of confidence intervals by Yang et al. (2021), but I do not think it is the correct tool for reasoning about prediction error for Bayesian models.

Thank you for your thoughtful comments and for highlighting the important distinction between EFI and Bayesian methods. Your observations regarding confidence interval construction and the role of prediction errors are well taken and largely correct.

To clarify some of the subtleties in how prediction error is handled, consider the following simple model:

where $\beta$ and $\sigma$ are model parameters, $Z_i \sim F$ represents random error from a distribution $F$, and $n$ is the sample size. Let $\theta = (\beta, \sigma)$ denote the full set of parameters. Suppose our goals are:

(i) to infer the function $g(x_0,\beta)$ at a given covariate $X=x_0$,
(ii) to predict a new response value $y_0$ at a given covariate $X=x_0$.

In Bayesian inference, if we have obtained posterior samples $\tilde{\theta}\_1, \ldots, \tilde{\theta}\_M$, then uncertainty of $g(x_0,\beta)$ is assessed via the variability in the samples $g(x_0,\tilde{\beta}\_1), \dots, g(x_0,\tilde{\beta}\_M)$ --- capturing epistemic uncertainty. For prediction (goal ii), aleatoric uncertainty is incorporated by generating

where $\tilde{Z}\_k \sim F$ are independent draws simulating the irreducible noise component (also independent of $Z_1,\ldots,Z_n$). Then prediction interval of $y_0$ can be constructed using $\tilde{y}\_{0,k}$, $k=1,2,\ldots,M$.

EFI, by contrast, draws samples from the fiducial distribution $\mu_n^*(\theta \mid D_n)$, based solely on the observed data $D_n=(X_1,y_1,\ldots,X_n,Y_n)$, without requiring a prior. This makes the inference more faithful to observations. Once fiducial samples of $\theta$ are obtained, the subsequent steps mirror those in Bayesian analysis: uncertainty in $g(x_0,\beta)$ is quantified through the empirical distribution of $g(x_0,\tilde{\beta}_1), \ldots, g(x_0,\tilde{\beta}_M)$, and predictive intervals for $y_0$ are constructed based on additional simulated noise $\tilde{Z}_k$, $k=1,2,\ldots,M$.

Notably, inference of the underlying true function $u_{\theta}(x)$ in the present paper corresponds to goal (i). It is also worth emphasizing that EFI adopts a fundamentally different approach to construct its fiducial distribution $\mu_n^*(\theta \mid D_n)$, which serves a similar role to the posterior distribution in Bayesian methods. Consequently, it necessitates a distinct sampling algorithm, differing from conventional MCMC techniques typically used in Bayesian inference.

Thank you for the advice, we reran the 1D–Poisson experiments (20 noisy boundary data points and 200 exact PDE data points) using dropout regularizers ranging from 0.1 to 0.0025, as summarized in Table 1. The highest coverage rate observed was 21.77%, which is substantially below the nominal 95% target. From the small MSE values, we see that Concrete Dropout provides accurate point estimates; however, the narrow CI widths and low coverage rates indicate that the interval estimates are not well-calibrated and tend to be overly narrow. As noted in [1], Concrete Dropout improves calibration compared to standard dropout, particularly through automatic tuning of dropout probabilities; however, it does not claim to fully resolve calibration challenges. In the context of PINNs, we suspect that the PDE regularizer — which requires computing gradients with respect to the network input — may interact with the dropout mechanism in a way that limits its effectiveness. This interaction may partly explain why the anticipated calibration benefits are not fully realized in our setting. Table1 Results for 1D-Poisson with Concrete Dropout

[1] Accurate Uncertainties for Deep Learning Using Calibrated Regression (https://proceedings.mlr.press/v80/kuleshov18a/kuleshov18a.pdf)