We thank the reviewer for acknowledging the effectiveness and theoretical rigor of our work. We appreciate their effort in helping us improve the efficiency and practicality of the work.

Computational costs.

As shown in Supp Table 7 of the paper, we have already compared the wall clock time between our RNP and the VI objectives and no significant differences were observed. Our computational complexity is linear to the number of MC samples, which is also comparable to the VI objective. We will add more clarifications for efficiency in sec 5.4.

Hyperparameter tuning of $\alpha$ and adaptive $\alpha$ tuning.

Since cross-validation can be computationally expensive, we have suggested the heuristics in sec 5.3 to gradually anneal $\alpha$ from 1 to 0 without hyperparameter tuning. We have already shown in Supp Table 5 that our RNP consistently outperformed competing approaches. We will elaborate further on this heuristic in sec 5.3.

Scalability on larger datasets or real-time applications.

We would like to clarify that NPs are generally scalable to large datasets due to minibatch training and fast inference of the framework, and we have tested all our NP models on a sizeable image dataset CelebA. To further improve our efficiency, we suggest to adopt variance reduction methods such as control variates and Rao-Blackwellization [1] that could require fewer MC samples. Additionally, for attention-based NPs, We suggest to use efficient attention mechanisms, e.g., Nyströmformer [2] which uses low rank approximation of the attention matrix. We will leave the efficiency improvement for our future work.

[1] Ranganath, R. et al, 2014, Black box variational inference. In AISTAT. PMLR.
[2] Xiong, Y. et al, 2021, Nyströmformer: A nyström-based algorithm for approximating self-attention. In AAAI.

$f$-divergence results.

We added the comparing results using $f$-variational bound in [3]. More specifically, by specifying a convex function $f$ and its dual $f^*$, $f$-divergence connects several divergences including KL and Rényi divergences. Based on eq (8) in [3], we have the objective $L_f(\phi, \theta) = \mathbb{E}_{q(z; \phi)}[f^*(\frac{p(z, Y_T|X_T, C; \theta)}{q(z; \phi)})] \geq f^* (p(Y_T|X_T, C))$.

We chose the posterior $q(z;\phi)= q(z|C, T; \phi)$ like NPs do and compared our RNP objective with two functions. $\chi^2$ divergence [3]: $f(u)= \frac{1}{u} - u, f^*(t)=t^2 -1$. and Jeffery divergence [4]: $f(u) = (u-1)\log u, f^*(t)=(t-1)\log t$.

The results provide additional support that the RNP objective improves the baseline models. We also tested f-divergence minimization approaches based on the bound of Nguyen et al [5] such as that proposed in [4]. However, as with many GAN-type objectives, they led to unstable training. Finally, unlike our approach, it is unclear how f-divergence minimization methods proposed in the literature can be extended to improve robustness of maximum likelihood-based NPs.

[3] Wan, N. et al, 2020, F-divergence variational inference. NeurIPS.

[4] Nowozin, S. et al, 2016, f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS.

[5] Nguyen, X. et al, 2010, Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory.

Intuition or visualizations for RNP posterior estimation.

In Fig 1 (a) we have visualized the two posterior distributions obtained by our RNP and VI objectives. The VI objective produced an overestimate of the posterior variance, indicating a strong regularization from the prior model. As a result, it fitted the periodic data with a large variance and over-smoothed mean functions as shown in Fig 1 (b), whereas RNP dampens the prior regularization and obtained a posterior with much smaller variance and therefore encourages the likelihood model to be more expressive as illustrated in Fig 1 (c).

We appreciate the reviewer's efforts in helping us refine the details and acknowledge the original literature. We agree that a rigorous analysis and self-sufficient figures would strengthen the soundness of our work.

Significance tests in Table 1 and 2.

All the results presented in the paper were reported with error bars (i.e., standard deviations). As the reviewer suggested, we performed two‐sample t-tests between our RNP and the second best method and highlighted the bold results with significant improvements of p value $<$ 0.05. We observed that in Table 1 we significantly improved RBF, Matern and Periodic datasets for most of the methods. Here we show the updated Table 2. In Table 2, the ML objective is no longer significantly superior than our RNP objective on the D_train EMNIST dataset.

Missing details for the Hare Lynx dataset.

We apologize for the missing details. Table 2 in the paper reported the results for both species as we treat them as a 2-dimensional input system with feature correlations (see sec 5.2). We left out the Hare plots in Fig 2 originally because when we tried to plot both the Lynx and Hare results in a single graph, the uncertainty intervals of two species were heavily overlapped and impeded the visibility. We will add the Hare results using different colors and opacity in the final version. We will also add more descriptions of this dataset in sec 5.2 and supplementary section for better clarification.

Relationship to hierarchical prior models.

We thank the reviewer for raising this interesting point. We believe our approach to misspecification in neural processes based on the Rényi divergence is fundamentally different to hierarchical approaches. By introducing additional latent variables in a hierarchical fashion, one aims to have a more flexible marginal prior model. For example, under some mild conditions, mixture models are known to be able to approximate any continuous density. However, usually, that additional flexibility comes at the cost of more complex inference. Our approach, maintains the original prior but deals with model misspecification through dampening with the $\alpha$ parameter. As pointed out by Reviewer Fmj6, it is more closely related to robust Bayesian inference methods based on Gibbs/tempered posteriors. We will discuss this in the final version.

Updating figures, captions, typos and references.

• Captions of Fig 3 and Fig 4. We will add captions for them to be self-sufficient. Fig 3: cross-validation is used to select the optimal $\alpha \in [0, 2]$. Fig 4: We investigated how MC sample sizes and the number of context points affect test log-likelihood.

• Uncertainty quantification in Fig 3. We have actually plotted the uncertainty intervals but some of them were occluded by thicker lines of mean values (see Fig 3 (a) TNPD $\alpha = 1.5$ for better visibility). We will change the color and the thickness of the intervals to make them more evident.

• $\alpha$ range and dynamic plot range in Fig 3. Fig 3 showed that $\alpha >1$ impedes NP training due to an overestimate of the posterior variance (please check the comment for Reviewer aNVZ ``The effect of $\alpha$ values on training'' for more details). Therefore, we recommend tuning $\alpha \in (0, 1)$ in the text. As suggested by the reviewer, we will highlight the results using the KL objective which corresponds to $\alpha = 1$ in the plot and change the plot into a dynamic range.

• Adding reference. We will add the original Rényi divergence paper for reference.

• Correcting typos. We will fix the typo in the limitation discussion and proof read the paper.

We thank the reviewer for acknowledging our "well-motivated theoretical derivations" and "strong empirical results".

Incremental novelty

We would like to clarify to the reviewer that our work goes beyond using RD for prior misspecification. We are the first to identify prior misspecification in the realm of NPs due to the parameter coupling between the prior conditional and the approximate posterior, and our theoretical and empirical analysis justify the use of robust divergences. Secondly, we introduce a new objective that unifies the VI and MLE objectives for NPs.

Adaptive $\alpha$ value selection

Please kindly refer to the comments for the Reviewer GkqA for this and f-divergence related questions.

Results on uncertainty calibration

We added the results of continuous ranked probability scores (CRPS) w.r.t different α values for NP and ANP on the RBF dataset. CRPS is a commonly used to measure the forecast accuracy and lower scores indicate better calibration. The results showed that the CRPS calibration improved when α increases from 0 to 0.9.

Theoretical discussions

We sincerely appreciate the reviewer's thoughtful feedback. While some of the questions raised by the reviewer extend beyond the immediate scope of this paper and address broader aspects of NP research, we recognize their importance and agree these points warrant further investigation. We are delighted to make every effort to clarify their concerns.

• Robust Bayesian inference and PAC-Bayes

Indeed, the literature on robust Bayesian inference is rich and we thank the reviewer for pointing out the need to discuss this. In particular, the relation to PAC-Bayes is fascinating. In short, dealing with model misspecification from a Bayesian perspective appears in the literature under several disguises, for example, Gibbs posteriors, tempered posteriors and fractional posteriors [1,2]. These approaches essentially weigh the likelihood to temper its influence. Connections with PAC-Bayesian bounds controlling generalization in statistical learning have also been explored previously, more notably in [2]. These connections are extended by explicitly analyzing variational approximations of PAC-Bayes/Gibbs posteriors [3]. We will expand on this in the final version. However, it is important to emphasize that our focus is on neural processes but these approaches remain exciting directions for future work.

[1] Grunwald, P, et al. Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 2017.

[2] Bhattacharya, A., et al. Bayesian fractional posteriors. The Annals of Statistics, 2019.

[3] Alquier, P., et al. On the properties of variational approximations of Gibbs posteriors. JMLR, 2016.

• Theoretical guarantees of RNP

The theoretical soundness of our work can be built on two parts: the frequentist consistency properties of the vanilla NPs using the KL divergence, and the relaxation of assumptions with our Rényi divergence. For the first part, we kindly refer the reviewer to section 3.3 -3.5 of the thesis [4] for prediction map approximation theory of NPs. Briefly, in the limit of infinite data, a variational family of prediction maps can recover the mean map of the true NP and the sum of the variance map and observation noise under some regularity assumptions. In the case of limited data, more assumptions including the input size, the compactness of the variational family, the boundedness of the data, and the boundedness of the stochastic process are required to guarantee the consistency. For the second part, some milder regularity conditions than KL divergence can be made for consistency [5], including a uniformly bounded prior distribution and a locally asymptotically normal likelihood model.

[4] Bruinsma, W. (2022) Convolutional Conditional Neural Processes. Apollo - University of Cambridge Repository.

[5] Jaiswal, P., Rao, et al, 2020. Asymptotic Consistency of α-Rényi-Approximate Posteriors. JMLR, 21(156).

• Data-dependent priors

We agree with the reviewer that it might sound un-Bayesian to have data-dependent priors, but in the case of NPs it is not only valid but necessary when viewed through the lens of hierarchical Bayes or meta-learning: the prior is being drawn from a hyper-prior. When we condition on context data, we're effectively doing Bayesian inference over a latent variable that defines the function.

Fixing typos and adding references

We will add the recommended references as well as [6] to the main text. We will also simplify some equation notations.

[6] Rodríguez-Santana et al, 2022. Function-space Inference with Sparse Implicit Processes. ICML.

We thank the reviewer for acknowledging our innovation of unifying the objectives and carefully reviewing details including supplementary materials and numerical results.

Additional evaluation metrics.

We have additionally reported the relative errors for two baseline methods NP and ANP on three GP regression datasets. Our objective still generally outperformed baseline models on this new metric. We will incorporate more results in the version.

The effect of $\alpha$ values on training.

$\alpha > 1$ usually impedes training as it focuses too much on improving the mass-covering of the posterior, resulting in an overestimate of the posterior variance (Please refer to Fig 3 in the paper for details). Regarding the convergence rate, [1] showed that under mild regularity assumptions, it is bounded by $\sqrt{n}$ with n being the number of samples rather than by $\alpha$ values.

[1] Jaiswal, P., Rao, et al, 2020. Asymptotic Consistency of $\alpha$-Renyi-Approximate Posteriors. JMLR, 21(156).

Missing clarifications and typos.

• Smoothness in Fig 1. The difference between L_RNP and L_NP on smoothness and correlation strength is only controlled by $\alpha$. Due to the misspecified prior regularization in standard KL, vanilla ANPs (Fig 1b) struggle with over-smoothed predictions. Our objective dampens such regularization and encourages a more expressive likelihood model that better fits the data.

• RNP advantage in Fig 2. RNP indeed did not impose a significant advantage over VI as shown in Fig 2 (a) and (b). However, our main claim in Fig 2 is that RNP achieves better uncertainty estimate than the ML objective.

• Typos and missing captions. We thank the reviewer for pointing out the typos in the captions of Fig 1, which we will correct in the revision. We will also add explanations for Fig 5, 6, 7 in the supplementary and fix the typos in sec 4 related work.