Uncertainty-Informed Meta Pseudo Labeling for Surrogate Modeling with Limited Labeled Data

Thank you for your constructive comments. We address each point below.

Q1: The core part of this work is the SSL, which just use the uncertainty as a tool.

A1: We thank the reviewer for the valuable comment. We agree that our work falls within the scope of semi-supervised learning (SSL), where uncertainty is employed to enhance pseudo-label quality. The central objective of UMPL is to improve model generalization to unseen data by introducing a role-aware dual uncertainty framework. This framework leverages epistemic uncertainty for teacher selection and aleatoric uncertainty for student feedback, forming a closed-loop refinement mechanism. Through this design, UMPL effectively exploits unlabeled data to enhance prediction accuracy in low-label scientific modeling settings. We will revise the manuscript to better highlight this focus.

Q2: Why don't just use most up to date uncertainty estimation methods or calibration methods?

A2: We appreciate the reviewer’s suggestion. Our main goal is to improve model accuracy in low-label settings by leveraging unlabeled data. While more advanced uncertainty estimators exist, they offer limited accuracy gains at significantly higher computational cost. UMPL adopts a balanced choice based on this trade-off, and already outperforms baselines (shown in Q3&A3). Moreover, UMPL is modular and can flexibly integrate more advanced teacher/student models if needed. We will clarify this design choice and comparative results in the revised version.

Q3: The proposed uncertainty estimation method is not well evaluated on widely-accepted benchmarks.

A3: We thank the reviewer for the helpful suggestion. Systematic comparisons of uncertainty estimation in PDE-based modeling are scarce—[1] is the only work that benchmarks multiple methods on the LE-PDE network to our best knowledge. To validate our approach, we conducted additional experiments on the public Darcy Flow dataset using AeroGTO, comparing UMPL with several uncertainty modeling baselines.

We use miscalibration area (MA) and root mean squared calibration error (RMSCE) as metrics to evaluate uncertainty quality. Results show that UMPL consistently achieves better uncertainty quality and predictive accuracy. We will include these results and clarify benchmarking choices in the revision. In addition, we replaced UMPL’s uncertainty components with more advanced estimators. While these alternatives achieved better calibration, they brought only marginal improvements in predictive accuracy and incurred significantly higher computational cost.

Q4: The computational overhead analysis is lacking.

A4: Thanks for the question. All experiments were conducted on a single NVIDIA RTX 4090 GPU. We report results on AeroGTO model across three representative tasks—2D steady-state, 2D transient, and 3D simulation—as summarized below.

Here, Supervised+uq refers to the student model used in UMPL. As shown, UMPL introduces almost no additional memory overhead, thanks to the REINFORCE-style update that avoids the costly backpropagation chain from the student to the teacher. This significantly reduces computational and memory complexity, at the cost of increased training time.

Q5: The sensitivity and robustness analysis.

A5: Thank you for the helpful suggestion. We evaluated sensitivity to aleatoric uncertainty on the OOD Stationary Lid-Driven Cavity dataset using AeroGTO as the backbone. Gaussian noise of varying magnitudes was added to the student’s predicted uncertainty during training.

The result shows that while uncertainty calibration degrades steadily with noise, the predictive accuracy remains stable under small perturbations. However, large noise leads to sharp performance drops, indicating that UMPL is robust to moderate noise but sensitive to severe distortion.

We also evaluated the sensitivity to the PUF threshold and feedback strength under the same setting:

A lower PUF helps filter noisy labels on harder datasets, though its impact is mild on this dataset. For feedback strength, too small values lead to no pseudo-label correction (reducing UMPL to plain PL), while overly large values inject noise, degrading label quality and student learning.

Reference

[1] Wu, Tailin, et al. "Uncertainty quantification for forward and inverse problems of pdes via latent global evolution." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 1. 2024.

[2] Liu, Q. and Wang, D., 2016. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29.

[3] Yashima, Shingo, et al. "Feature space particle inference for neural network ensembles." International Conference on Machine Learning. PMLR, 2022.

[4] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems 30 (2017).

Thank you for your careful reading of the paper and your constructive comments. We address each point below.

Q1:Request to clarify and support the claimed benefits of role-aware uncertainty modeling and targeted feedback.

A1: In response to the request for evidence supporting the claims of “role-aware design” and “targeted feedback,” we provide the following clarifications:

Regarding the second bullet point: “Role-aware design leads to more robust and interpretable uncertainty estimates” — our framework uses role-specific uncertainty: the teacher models epistemic uncertainty for distribution shifts, while the student captures aleatoric uncertainty for input-dependent noise, forming a complementary structure. This design is supported by:

• Theoretical justification: Section 3.1 provides the theoretical foundation, where Theorem 3.1 introduces EMC Dropout for efficient epistemic uncertainty estimation on the teacher side, Equation (5) formulates the aleatoric uncertainty optimization for the student, and Theorem 3.2 establishes an upper bound on student error under noisy pseudo labels.

• Empirical evidence: Figure 2 shows the student’s feedback loss aligns well with true error, demonstrating strong spatial adaptivity. Figures 3(a) demonstrates that our method, applied to the Ahmed dataset, provides more accurate uncertainty estimates compared to the approach that uses only the student model trained on labeled data.

Regarding the third bullet point: “Targeted feedback significantly improves the quality of pseudo labels” This is supported both theoretically and empirically:

• Theoretical justification: Section 3.2 introduces a point-wise feedback mechanism guided by the student’s aleatoric uncertainty. The pseudo labels are perturbed (Eq. 6), and the teacher is updated using a first-order gradient approximation (Eq. 10–12), which avoids expensive meta-gradient computation.

• Empirical evidence: Figure 2 shows that feedback signals correlate strongly with true label error, guiding the teacher to focus on high-error regions and improving pseudo-label quality over time. Tables 1 and 2 demonstrate that UMPL consistently outperforms methods without targeted feedback (e.g., Pseudo Label, Mean Teacher), and Table 3 shows a 42.6% performance drop when the feedback module is removed, confirming its critical role.

We acknowledge that the current version lacks direct quantitative analysis of pseudo-label quality progression. In the revised version, we will include comparisons of pseudo-label error curves over iterations to more explicitly demonstrate how targeted feedback improves pseudo-label quality.

Q2:Why to choose relative L2 error?

A2: Thank you for the question. We adopt relative L2 error due to its clear advantages in scientific modeling, particularly in the following aspects:

• It is well-suited for multi-scale systems or quantities with heterogeneous units, where standard L2 loss tends to overweight large-magnitude regions and ignore small but important ones.

• In multi-physics problems (e.g., Plasma ICP), where predicted variables differ by orders of magnitude, standard L2 loss leads to gradient imbalance. Relative L2 provides scale-invariant optimization that treats all fields fairly.

• This choice is widely adopted in neural operator literature (e.g., FNO, Transolver, AeroGTO), both as a training loss and evaluation metric, due to its physical interpretability and robustness.

Therefore, relative L2 error not only serves as an effective evaluation metric but also as a more appropriate training objective.

Q3:Why to choose an encoder embedding? Is this a popular choice in the student-teacher literature? Is it work better for modeling PDE systems?

A3: Thank you for the question. The use of encoder embeddings is motivated by the following:

• Unified representation of heterogeneous PDE inputs: PDE problems involve geometry, physical parameters, and boundary conditions. The encoder maps these multimodal inputs into a common latent space, enhancing generalization. For instance, in the Plasma ICP case, the encoder helps integrate irregular mesh geometry and spatial parameters into a consistent representation.

• Widely used across domains: Teacher–student frameworks commonly use encoder embeddings in tasks such as image classification[1] and defect detection[2], enabling stable pseudo-labeling and alignment between models.

• Adaptation to PDE-specific challenges: For datasets with non-uniform meshes or complex geometries (e.g., Darcy Flow, NSM2d), the encoder learns geometry-invariant representations to improve spatial robustness. This strategy is also used in operator models like DeepONet and AeroGTO to align heterogeneous inputs into a unified latent space.

We will clarify this design choice in the revised version.

Q4: Figure 2 isn't clear which method is better.

A4: Thank you for the valuable feedback. In the revised version, we will replace the last subplot in Figure 2 with a comparison of Kendall correlation between feedback loss and true pseudo-label error for UMPL and Noisy TS. On 100 unlabeled samples, UMPL achieves much stronger correlation: 59 samples exceed 0.8 (vs. 5 for Noisy), and only 2 fall below 0.6 (vs. 11 for Noisy). This highlights that UMPL’s aleatoric uncertainty-guided feedback more reliably captures pseudo-label error, leading to more stable and robust training.

Q5: Comments regarding the REINFORCE rule.

A5: Thank you for the question. We adopt REINFORCE to estimate the gradient in Eq.(8), primarily to avoid the costly backpropagation path from student to teacher, thus significantly reducing memory and computational overhead. REINFORCE is a standard technique and has been widely used in meta pseudo-labeling methods in computer vision. Its main limitation is increased time cost due to repeated forward and backward passes. We will clarify this design choice and trade-off in the revised version.

Q6: Why not choose other uq methods?

A6: Thank you for the insightful question. As noted, we chose EMC mainly for its computational efficiency. While more accurate uq methods exist, EMC offers a good balance between cost and performance. Our main goal is to use unlabeled data to boost prediction accuracy on unseen data. We have added experiments comparing different uq methods on AeroGTO for Darcy Flow dataset in terms of both accuracy and efficiency, using miscalibration area (MA) and root mean squared calibration error (RMSCE) as evaluation metrics:

As shown in the results, the teacher performs worse than advanced Bayesian methods (e.g., SVGD, feature-WGD), and the student underperforms compared to Ensemble latent-uq. However, both are more efficient. As discussed in Q7&A7, Advanced replacements improve uncertainty estimation with minimal accuracy gains, at a much higher computational cost. Our framework remains flexible, enabling such substitutions depending on practical trade-offs.

Q7: Other choices for the building blocks of UMPL?

A7: We replaced the student and teacher components using the same setup as in Q6&A6. The performance comparison is shown below:

These results confirm that replacing components leads to better performance, at the cost of increased computational overhead.

Q8: Computational resources & statistical significance?

A8: Thanks for the question. All experiments were conducted on a single NVIDIA RTX 4090 GPU. We report results on AeroGTO model across three representative tasks—2D steady-state, 2D transient, and 3D simulation—as summarized below.

Due to main text space constraints, we reported only the mean over 5 trials with different seeds. Statistical significance results will be included in the appendix of the revised version.

Q9: Typo in Appendix H title.

A9: Fixed it.

Reference

[1] Wang, Zhenbin, et al. "Metateacher: Coordinating multi-model domain adaptation for medical image classification." Advances in Neural Information Processing Systems 35 (2022): 20823-20837.

[2] Zhao, Sinong, et al. "Meta pseudo labels for anomaly detection via partially observed anomalies." Engineering Applications of Artificial Intelligence 126 (2023): 106955.

[3] Yashima, Shingo, et al. "Feature space particle inference for neural network ensembles." International Conference on Machine Learning. PMLR, 2022.

[4] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems 30 (2017).

[5] Wu, Tailin, et al. "Uncertainty quantification for forward and inverse problems of pdes via latent global evolution." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 1. 2024.

Thank you for the positive feedback.

Q1: Quantitative proof that the predicted uncertainty reliably correlates with error magnitude.

A1: Thank you for the insightful question. We have added experiments comparing our method with other uncertainty estimation approaches on AeroGTO for the Darcy Flow dataset in terms of both accuracy and efficiency, using miscalibration area (MA) and root mean squared calibration error (RMSCE) as evaluation metrics:

Additionally, we conducted a comparison on the NSM2d experiment:

These comparisons highlight UMPL's strong performance over baselines with a good balance between accuracy and efficiency. The teacher and student components can be flexibly replaced with more advanced methods for further gains, albeit at higher computational cost.

Q2: The method may not hold up with the noise or real-world more complex data.

A2: Thank you for the thoughtful comment. We have considered challenging scenarios in our experiments. For example, the Black Sea dataset is based on real ocean measurements with strong seasonal patterns and complex temporal evolution. It poses a difficult temporal OOD generalization problem, yet UMPL still achieves performance gains (albeit smaller than on other datasets). The Plasma ICP dataset involves distribution shift from low- to high-fidelity simulations. UMPL outperforms baselines using only 10% of high-fidelity data, showing strong data efficiency under realistic conditions.

We acknowledge that our benchmarks do not include noisy data. The current teacher model focuses on epistemic uncertainty and does not explicitly model data noise. To address this, we conducted an additional experiment on the Darcy Flow dataset using the AeroGTO backbone, introducing varying levels of input-dependent noise.

Results show that model performance degrades with added noise. To improve robustness with noise, we incorporated both epistemic and aleatoric uncertainty in the teacher following [5], and added an ensemble student model. This enhanced predictive performance under noisy conditions, which we will detail further in the revised version.

Q3: How does UMPL handle physical systems with asymmetric or heavy-tailed uncertainties?

A3: Thank you for the insightful question. Our teacher and student models indeed assume Gaussian scenarios, which is suitable for most standard cases. However, we acknowledge that performance may degrade under shocks or discontinuities. To evaluate this, we designed an experiment using a 1D Burgers-type PDE with small diffusion coefficient:

$\frac{\partial u}{\partial t}-\mu \frac{\partial ^2 u}{\partial x^2}+u \frac{\partial u}{\partial x}+\epsilon u^3=0,u_0(x)=ax^3-(a+1)x,x\in[-1,1],-\mathbf{n}\cdot\frac{\partial u}{\partial x}|_{x=-1,1}=0$

Using DeepONet as the backbone, results show that the original model struggles under such non-Gaussian settings:

We improved performance by replacing the teacher's epistemic model with feature-WGD and the student's aleatoric model with MDN [6], which better capture heavy-tailed distributions. This comes at higher computational cost. Our framework thus offers a flexible trade-off between efficiency and accuracy, with modular components adaptable to specific scenarios. We will include these findings in the revised appendix.

Q4: Could you report computing resources?

A4: Thanks for the question. All experiments were conducted on a single NVIDIA RTX 4090 GPU. We report results on AeroGTO model across three representative tasks—2D steady-state, 2D transient, and 3D simulation—as summarized below.

Here, Supervised+uq refers to the student model used in UMPL. As shown, UMPL introduces almost no additional memory overhead, thanks to the REINFORCE-style update that avoids the costly backpropagation chain from the student to the teacher. This significantly reduces computational and memory complexity, at the cost of increased training time.

Reference

[1] Liu, Q. and Wang, D., 2016. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29.

[2] Yashima, Shingo, et al. "Feature space particle inference for neural network ensembles." International Conference on Machine Learning. PMLR, 2022.

[3] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems 30 (2017).

[4] Wu, Tailin, et al. "Uncertainty quantification for forward and inverse problems of pdes via latent global evolution." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 1. 2024.

[5] Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." Advances in neural information processing systems 30 (2017).

[6] Thakur, Akshay, and Souvik Chakraborty. "MD-NOMAD: Mixture density nonlinear manifold decoder for emulating stochastic differential equations and uncertainty propagation." arXiv preprint arXiv:2404.15731 (2024).

Dear Reviewer 9hkT,

We have provided new experiments and discussions according to your valuable suggestions, which have been absorbed into the revised manuscript. We hope that the new manuscript is made to be stronger with your suggestions.

As the rebuttal deadline is approaching, we sincerely look forward to your reply. Thanks so much for your time and effort!

Best Regards,

Authors of Submission 10749

Thank you for your positive feedback and for recognizing the significance and potential of our method.

Q1: Clarity and presentation – Several aspects of the manuscript hinder readability.

A1: We thank the reviewer for the helpful suggestions. For a and b, we will include key variable definitions and move Algorithm 1 to the main text in the revised version. For c, we will revise Figure 1 in the revised version to better reflect the complete UMPL workflow. The updated figure will explicitly show the teacher–student interaction, the three-stage training loop (pseudo-label generation, student update, teacher update), and the roles of Progressive Uncertainty Filtering (PUF) and Supervised Anchoring (SA), thereby enhancing the clarity and interpretability of the framework.

Q2: Could you elaborate on the computational resources your method requires?

A2: Thanks for the question. All experiments were conducted on a single NVIDIA RTX 4090 GPU. We report results on AeroGTO model across three representative tasks—2D steady-state, 2D transient, and 3D simulation—as summarized below.

Here, Supervised+uq refers to the student model used in UMPL. As shown, UMPL introduces almost no additional memory overhead, thanks to the REINFORCE-style update that avoids the costly backpropagation chain from the student to the teacher. This significantly reduces computational and memory complexity, at the cost of increased training time.

Q3: Within the teacher–student framework, is there a risk of model collapse? How to protect against this?

A3: We thank the reviewer for the insightful question. We agree that model collapse is a potential risk in teacher–student frameworks. To mitigate this, our design incorporates several mechanisms:

• Divergent Initialization Strategy: The teacher and student models are initialized with different parameters, and during each training loop, they are exposed to independently sampled labeled data. This helps reduce the risk of early-stage over-alignment due to shared errors.

• Uncertainty-Guided Feedback Loop: The student provides point-wise feedback based on aleatoric uncertainty, which captures spatial error sensitivity and directs the teacher’s updates toward high-error regions. This breaks blind imitation and reduces the chance of reinforcing incorrect signals.

• Uncertainty-Aware Pseudo Labels: Pseudo labels are perturbed using the teacher’s epistemic uncertainty, replacing hard targets with uncertainty-aware guidance. This alleviates label bias and stabilizes convergence.

• Progressive Uncertainty Filtering (PUF): Before being used for training, the teacher’s pseudo labels are filtered by uncertainty. Low-confidence samples are discarded, improving pseudo-label quality and enhancing training stability.

We will include a detailed discussion of these safeguards in the revised version to better address this potential risk.

Dear Reviewer fwWx,

We have provided new discussions according to your valuable suggestions, which have been absorbed into the revised manuscript. We hope that the new manuscript is made to be stronger with your suggestions.

As the rebuttal deadline is approaching, we sincerely look forward to your reply. Thanks so much for your time and effort!

Best Regards,

Authors of Submission 10749