Learning to Generalize: An Information Perspective on Neural Processes

Thanks for your positive and valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.

Q1. Novelty of Noise Injection Strategy

Reply: We establish a novel generalization bound for Neural Processes that connects their performance to the mutual information $I(\theta; D_{1:m})$ between model parameters and training tasks. This bound reveals that tighter generalization requires controlling both the complexity of learned representations and their sensitivity to task variations. Based on this theoretical foundation, we systematically derive two complementary components: DSR regularization that directly constrains $I(\theta; D_{1:m})$ through Hessian properties, and adaptive noise injection $\sigma^2_s = \eta_s/\gamma$ that enhances optimization trajectory stability across task distributions. Unlike SGLD's thermodynamic sampling or entropy-SGD's diversity-seeking exploration, our noise injection is contextually modulated based on meta-learning requirements, targeting the uncertainty inherent in generalizing across diverse task families. This creates an adaptive mechanism that responds to meta-learning dynamics rather than applying uniform perturbations, making it particularly effective for few-shot learning scenarios where models must balance expressiveness with generalization across limited data.

Q2. Direct Empirical Validation of Flatter Minima

Reply: We conducted additional experiments to directly demonstrate that our method achieves flatter minima. We computed Hessian-based curvature measures including Hessian Trace (sum of eigenvalues indicating overall curvature) and Maximum Eigenvalue (capturing the sharpest direction) using finite differences approximation around converged parameters on 1D regression with RBF kernels across 5 random seeds. Table 1 clearly shows that our Gen-variants consistently achieve significantly lower curvature values across all measures, providing direct empirical evidence that our method finds flatter minima. We will add intuitive visualization plots showing loss landscape contours and optimization trajectories in the revised version to provide clearer visual evidence of the flatter minima achieved by our approach.

Table 1: Hessian-based Curvature Measures

Q3. Connection to Meta-Learning and Benchmark Comparisons

Reply: The connection between meta-learning and flatter optima lies in the generalization hierarchy of meta-learning. Traditional learning optimizes for single-task performance, while meta-learning must generalize across task distributions. Flatter minima in meta-parameter space correspond to more robust representations that are less sensitive to task-specific variations, which is crucial for few-shot adaptation. Neural Processes must encode prior knowledge about function families while remaining flexible for new tasks, and our approach specifically targets this by regularizing learning dynamics toward solutions stable across task variations. To address benchmark comparisons, we extended our evaluation to other meta-learning methods, showing consistent improvements across different paradigms beyond just Neural Processes. We compared against MAML[1], a gradient-based meta-learning algorithm that learns optimal initialization parameters for rapid adaptation, and MALIBO[2], a representative Bayesian meta-learning method that uses probabilistic models to capture task uncertainty, demonstrating that our approach provides consistent benefits across diverse meta-learning frameworks.

Table 2: Meta-learning method comparison (MAE)

Q4. Comparison with Conventional Optimization Strategies

Reply: Our approach operates on a different technical pathway from conventional strategies like learning rate scheduling, weight decay, and dropout - they can coexist without conflict since they target different aspects of optimization. Both our Gen-NP variants and baseline methods use identical conventional optimization setups including Adam optimizer with L2 regularization (weight_decay=1e-4), cosine annealing scheduling, and gradient dropout (rate 0.1). Since all methods share the same optimization foundation, the observed improvements directly result from our DSR regularization and adaptive noise injection components. The key difference is that conventional methods provide general regularization, while our approach offers task-distribution-aware regularization through adaptive noise scaling $\sigma^2_s = \eta_s/\gamma$ that specifically targets meta-learning dynamics. This demonstrates that our method adds meaningful value beyond existing techniques by addressing the unique challenges of Neural Processes in few-shot learning scenarios.

Q5. Reparameterization-Invariant Sharpness Metrics

Reply: You raise an excellent point about reparameterization sensitivity. To address this limitation, we implemented two reparameterization-invariant metrics as shown in Table 3. Effective Sharpness[3] normalizes perturbations by adaptive scaling factors ($\hat{\epsilon} = \epsilon / \max(|\theta|, \gamma)$ with $\gamma = 1e-3$, $\rho = 0.05$), while Distribution Sharpness directly measures output sensitivity to parameter perturbations ($\epsilon = 0.01$). Both metrics are computed every 50 training steps using fresh data batches. Table 3 demonstrates that our Gen-variants achieve substantial reductions across both invariant metrics compared to baselines. This confirms that the observed flatness represents genuine improvements in loss landscape geometry rather than artifacts of parameter scaling, validating that our approach achieves meaningfully flatter minima robust to different measurement methodologies.

Table 3: Reparameterization-Invariant Sharpness Metrics

References

[1] Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML).
[2] Pan J, Falkner S, Berkenkamp F, et al. (2024). MALIBO: Meta-learning for likelihood-free Bayesian optimization. In Proceedings of the 41th International Conference on Machine Learning (ICML).
[3] Kwon, J., Kim, J., Park, H., & Choi, I. K. (2021). ASAM: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning (ICML).

Thanks for your positive and valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.

Q1. Method novelty and innovation

Reply: We appreciate the reviewer's concern about method novelty and the structural derivation of our approach. Our principal contribution is establishing a theoretical foundation that connects generalization performance of Neural Processes with parameter space geometry through information theory. Unlike ad-hoc combinations of existing techniques, our DSR regularization and noise injection strategy are systematically derived from analyzing the generalization bound structure. The DSR term directly constrains the mutual information $I(\theta; D_{1:m})$ between model and data by controlling Hessian properties, while adaptive noise injection $\sigma^2_s = \eta_s/\gamma$ enhances optimization trajectory stability. While individual components like sharpness regularization exist, our unified information-theoretic framework specifically addresses the hierarchical optimization challenges unique to Neural Processes, handling both task-level and sample-level complexities simultaneously. This theoretical guidance provides principled design choices rather than empirical combinations, distinguishing our approach from alternatives like dropout or other stochastic regularization methods.

Q2. Mutual information evaluation

Reply: Thanks for this excellent suggestion about empirically validating our theoretical claims. Since direct computation of mutual information $I(\theta; D)$ is computationally intractable, we initially followed the approach[1] and used gradient inconsistency (GI) to measure mutual information, with results reported in our paper. Based on your suggestion, we implemented another gradient-statistics-based method to approximate mutual information, which computes MI through gradient variance across different tasks, gradient correlations between task pairs, and gradient entropy estimation. Table 1 clearly shows that our Gen-NPs consistently achieve lower mutual information $I(\theta; D)$ compared to baseline methods, which validates our theoretical framework. The results demonstrate that Gen-NPs reduce both mutual information and gradient inconsistency across all model variants, confirming that our approach successfully tightens the generalization bound as predicted by theory.

Table 1: Mutual Information and Gradient Inconsistency Comparison on 1D Regression Task

Q3. Formatting issues

Reply: Thank you for pointing out the formatting errors. We sincerely apologize for these oversights and will conduct a thorough review of the entire manuscript to ensure all formatting, references, and notations are correct in the final version.

Q4. Hyperparameter selection

Reply: Thanks for asking about hyperparameter selection. We conducted comprehensive grid search analysis to justify our $\lambda_1$ and $\lambda_2$ ranges. Table 2 shows the systematic evaluation across multiple parameter combinations demonstrates that our suggested ranges $[0.01, 0.1]$ for $\lambda_1$ and $[0.001, 0.01]$ for $\lambda_2$ provide robust performance with optimal configurations achieving consistent improvements. The grid search results show that performance remains stable within reasonable parameter ranges, with optimal region ($\lambda_1 \in [0.03, 0.07]$, $\lambda_2 \in [0.003, 0.007]$) achieving $<2\%$ variance. While these ranges are somewhat problem-dependent, our extensive experiments across different kernels and tasks suggest they generalize well to various Neural Process applications.

Table 2: $\lambda_1$ and $\lambda_2$ sensitivity analysis (Gen-NP MAE results on RBF kernel with $\gamma=1\text{e}7$)

Q5. Statistical significance of improvements

Reply: We acknowledge that LL and GI improvements appear modest compared to Bayesian Optimization and Contextual Bandits results. To address whether these improvements are statistically meaningful, we conducted rigorous significance analysis using paired t-tests across 10 independent runs with different random seeds. Table 3 shows that all improvements are statistically significant with p-values well below 0.05, confirming these are reliable enhancements rather than random fluctuations. The key insight is that small numerical differences can be highly significant when experimental variance is low and improvements are consistent across runs. In regression tasks, even modest enhancements in likelihood and uncertainty quantification translate to substantial practical benefits for applications requiring reliable probabilistic predictions. The larger improvements in BO and Contextual Bandits reflect the different nature of these tasks, where our method's ability to escape sharp minima and find flatter solutions has more pronounced effects on optimization trajectories and decision-making performance.

Table 3: Statistical Significance Analysis (p-values from paired t-tests)

All p-values are well below 0.05, confirming statistical significance with high confidence levels.

References

[1] Chen Q, Shui C, Marchand M. Generalization bounds for meta-learning: An information-theoretic analysis[J]. Advances in Neural Information Processing Systems, 2021, 34: 25878-25890.

Thanks for your positive and valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.

Q1. Method novelty and innovation (Weakness 1 & 2)

Reply: We appreciate the reviewer's concern about novelty and innovation. Our principal contribution is to establish a theoretical foundation connecting generalization performance of Neural Processes with parameter space geometry with the aid of information theory. By analyzing its generalization bound, two synergistic components (DSR regularization and noise injection) can be systematically derived. Among them, DSR term directly constrains the mutual information $I(\theta; D_{1:m})$ between model and data by controlling Hessian properties, while the adaptive noise injection $\sigma^2_s = \eta_s/\gamma$ enhances the optimization trajectory stability. As we known, this is the first unified information-theoretic framework to handle with both task-level and sample-level complexities for the hierarchical optimization structure of Neural Processes.

Q2. Computational complexity analysis and overhead (Weakness 3 & Question 1)

Reply: Yes, as you mentioned and shown in Appendix D, the computational complexity is slightly higher than the original baseline. Even though we introduce additional components, they can be efficiently implemented via Hutchinson estimators for $\text{Tr}(H)$ approximation and optimized auto-differentiation. Table 1 demonstrates that modest training overhead (10-18%) yields substantial performance improvements (4-8% in log-likelihood), making Gen-NPs highly practical for applications requiring reliable uncertainty quantification. Table 2 shows the theoretical complexity comparison where our additional operations scale linearly with model parameters.

Table 1: Practical computational overhead comparison

Table 2: Theoretical complexity comparison between standard NPs and Gen-NPs

Q3. Hyperparameter sensitivity analysis (Question 2)

Reply: We conducted comprehensive sensitivity analysis on the 1D regression task with RBF kernels. The hyperparameters $\lambda_1$ and $\lambda_2$ control the relative importance of context and target loss variance components in our DSR regularization. Based on empirical analysis, we set $\lambda_1 \in [0.01, 0.1]$ and $\lambda_2 \in [0.001, 0.01]$ to maintain appropriate balance between the two terms, where $\lambda_1$ typically takes larger values as context loss variance generally has greater impact on meta-learning stability. Table 3 demonstrates that our method exhibits good robustness to hyperparameter variations, with performance remaining stable within these reasonable parameter ranges, indicating that the method does not require intensive hyperparameter tuning for practical deployment. Table 4 shows the noise scaling parameter sensitivity analysis. We will include sensitivity analysis results for additional experiments in the revised version.

Table 3: $\lambda_1$ and $\lambda_2$ sensitivity analysis (Gen-NP MAE results on RBF kernel with $\gamma=1\text{e}7$)

Table 4: Noise scaling parameter $\gamma$ sensitivity analysis with $\lambda_1=0.05$ and $\lambda_2=0.005$

Q4. Extension to other meta-learning algorithms (Question 3)

Reply: Our framework demonstrates excellent extensibility due to the universal nature of generalization challenges in meta-learning. Actually, Neural Processes is one kind of model-specific meta-learning methods by learning task representations through context encoding. (Note that MAML[1] represents model-agnostic meta-learning through gradient-based adaptation.) Both model-specific and model-agnostic methods are hard to learn the generalizable representations from limited samples. In this situation, the proposed information-theoretic framework are suitable for both categories of meta-learning. Table 5 shows the experimental results on MAML and MALIBO[2] (for 1D regression task with RBF kernels) further demonstrate the consistent improvements across different meta-learning paradigms.

Table 5: Meta-learning method comparison (MAE)

Q5. Apparent contradiction between noise effects and Tr(H) (Question 4)

Reply: This question touches on a fundamental principle in modern optimization theory: the connection between loss surface sharpness and generalization performance. Following the established framework from sharpness-aware minimization research[3][4], flatter minima in the loss landscape correspond to better generalization, as they represent solutions less sensitive to parameter perturbations. Our approach operates on this principle by using noise injection as an exploration mechanism to escape sharp, locally optimal regions, while the DSR regularization term $\lambda_1 \cdot \text{Tr}(H) + \lambda_2 \cdot \|H\|_F$ actively guides the optimization toward flatter regions with lower curvature. The apparent contradiction dissolves when we recognize that momentary increases in local curvature during noise-driven exploration are fundamentally different from the global optimization objective of minimizing the generalization bound. The noise helps us discover flatter minima that we would not reach through standard gradient descent, resulting in improved generalization despite temporary local perturbations.

References

[1] Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML).
[2] Pan J, Falkner S, Berkenkamp F, et al. (2024). MALIBO: Meta-learning for likelihood-free Bayesian optimization. In Proceedings of the 41th International Conference on Machine Learning (ICML).
[3] Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2021). Sharpness-aware minimization for efficiently improving generalization. In Proceedings of the 9th International Conference on Learning Representations (ICLR).
[4] Kwon, J., Kim, J., Park, H., & Choi, I. K. (2021). ASAM: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning (ICML).