We thank the reviewer for acknowledging our theoretical novelties in the ML-based objectives and the comprehensive experiments.

Prior misspecification motivation and experiment validation.

The prior misspecification defined in Proposition 3.2 is the mismatch between the model prior distribution $q(\mathbf{z}|C, \varphi)$ which is conditioned on BOTH the parameters $\varphi$ and context data ${C}$ and the ground truth prior distribution p(\mathbf{z}|\{C}, \varphi^*) whose ground truth parameters $\varphi^*$ are unknown (which are not the training/testing distributions the reviewer was mentioning). We can risk obtaining a misspecified prior by conditioning either on a poor parameterization or poor data, which is what we've covered in the experiment Section 7.1 and section 7.2.

Scenario 1: Poor parameterization $q(\mathbf{z}|{C}, \varphi_{bad})$ (Section 7.1)

meaning NN parameters $\varphi_{bad}$ are far from the ground truth values $\varphi^*$. This can happen in NPs because their posterior and prior models are forced to couple the parameters, which is why we saw performance improvements on Table 1 with the Renyi divergence even when there are no domain shifts.

Scenario 2: Poor data $q( \mathbf{z}|{C}_{bad}, \varphi*)$ (Section 7.2)

meaning the data $C_{bad}$ are sampled from a different distribution than training. This, as the reviewer pointed out, can be caused by additional noise or uncontrollable factors. We used the domain shift data sets Lotka-Volterra and EMNIST to validate the performance of the model, as we can ensure that the data distribution is different, and the results in Table 2 have validated the superiority of the RNPs in that regard. Our framework can indeed handle both types of misspecification with the Renyi Divergence by mitigating the impact of the prior model $q(z|C_{bad}, \varphi_{bad})$ on the posterior and likelihood model with the alpha value and it does not distinguish the two scenarios. We have added clarifications in both the definition and experiments.

Other points.

1. Target dependencies in ANP. Target sets are indeed assumed independent in ANPs and their attention is computed within the context set or between the context and each target point. We have moved this part to the posterior model for introducing dependencies. We also added the introduction of dependencies of GNPs in the ML objective section.

2. Autoregressive TNP. We have clarified TNP-A to be the model which incorporate dependencies.

3. Posterior typo. We have corrected the typo to $q(\mathbf{z}|X_T, Y_T, X_C, Y_C)$.

4. Reference. We have referred to the original TNP paper.

Q1 VI/ML objective for RNPs.

We adopted the VI objectives for NPs, ANPs and VNPs (Versatile Neural Processes) as their model designs include the prior models. We used the ML objective for TNP and BANP(Bayesian aggregation NPs) because the TNP objective was originally defined using ML only and the ML objective significantly outperformed the VI objective for BANPs. We have clarified the setting in the experiment section.

Q2 Reporting context log-likelihood.

Our reported value is based on the overall predictive distribution including both the context and target sets: $p(Y_T, Y_C|X_T, C)$. It's important to distinguish between context and target log-likelihoods because, during training, Neural Process models can sometimes improve overall log-likelihood by sacrificing one of these aspects. Similar to [3,4], separately checking context and target log-likelihoods would provide more insights.

[3] J. Lee, Y. Lee, J. Kim, E. Yang, S. J. Hwang, and Y. W. Teh. Bootstrapping neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.

[4] H. Lee, E. Yun, G. Nam, E. Fong, and J. Lee. Martingale posterior neural processes. In International Conference on Learning Representations (ICLR), 2023.

Q3 EMNIST dataset performance.

No hyperparameter tuning was adopted on the alpha value, which is possible to obtain suboptimal results than the original ML objective. EMNIST dataset seems to produce a higher variance in the test log-likelihood 1.47±0.12 than the datasets in Table I (usually ~0.0x) which could contributes to the worse results in the testing set with no domain shifts.

We thank the reviewer for clarifying the questions for discussion.

Scenario 1

The objective of NP in eq(3) suggests that by coupling the parameters $\varphi$ the model can minimize $KL(q(z|C, T, \varphi) \mid q(z|C, \varphi))$, for example, by moving the prior $q(z|C, \varphi)$ closer to the posterior $q(z|C, T, \varphi)$, but it doesn't guarantee the minimization of $KL(q(z|C, T, \varphi)\mid p(z| C, \varphi^*))$ which corresponds to maximizing the true ELBO.

Scenario 2

We have added a context-target mismatch experiment in Supplementary Table 4. The setting is detailed as follows: during both training and testing, we keep the target data $(X_T, Y_T)$ clean and corrupt the context data with noise $\tilde{y} = (1-\beta) * y_C + \beta * \epsilon, \epsilon \sim \mathcal{N}(0, 1)$. The noise level $\beta$ is set as 0.3 for both training and testing. Therefore, the marginal predictive distribution is $p(Y_T|X_T, X_C, \tilde{Y})$. This can indeed be categorized as the poor context $p(z|C_{bad}, \varphi)$. Our method still showed outperformance across multiple datasets as the impact of misspecified contexts is alleviated via the divergence. Along with the train-test distribution shift scenario, which the reviewer believes to be categorized as bad parameterization, we have shown improvements in both scenarios. We have added relevant clarifications in section 7.2 to distinct the two scenarios.

Hopefully our responses have helped address your concerns. We are more than happy to provide further discussions if needed.

We deeply appreciate the reviewer's acknowledgment with our experimental validation and outperformance. We will start by addressing the more important questions.

W2 Analytical analysis of prior misspecification.

We can write the gradient of the objective (6) wrt the parameters of the posterior as

$\mathcal{L} = \frac{1}{1-\alpha} \log E_{q(\mathbf{z}|C, T, \varphi )} \left[\frac{p(Y_T| X_T, \mathbf{z}, \theta) q(\mathbf{z}|C, \varphi)}{q(\mathbf{z}|C, T, \varphi)}\right]^{1-\alpha}$

$\nabla_\varphi \mathcal{L} = E_{q(\mathbf{z}|C, T, \varphi)} \left[ w_\alpha(\mathbf{z}, X_T, Y_T, C, ) \nabla_\varphi \log \frac{p(Y_T|X_T, \mathbf{z}, \theta) q(\mathbf{z}| C, \varphi)}{q(\mathbf{z}|C, T, \varphi)}\right]$

with $w_\alpha(\mathbf{z}, X_T, Y_T, C) = \left(\frac{p(Y_T|X_T, \mathbf{z}, \theta) q(\mathbf{z}|C, \varphi)}{q(\mathbf{z}|C, T, \varphi)} \right) ^{1-\alpha} / {E}_{q(\mathbf{z}|C, T)} \left[ \frac{p(Y_T|X_T, \mathbf{z}, \theta) q(\mathbf{z}|C, \varphi)}{q(\mathbf{z}|C, T, \varphi)}\right]^{1-\alpha}$

$w_\alpha(\mathbf{z}, X_T, Y_T, C)$ is the normalized importance weights. Therefore, the gradient of RNP equals the gradients of the VI objective scaled by the weight $w_\alpha(\mathbf{z}, X_T, Y_T, C)$. If the prior model $p_\varphi(\mathbf{z}|C)$ is misspecified, by selecting different $\alpha$ values, we can control the behavior the posterior update which consequently affect the posterior variance and mitigate prior misspecification. In a simple case of a Bayesian linear regression model $y= \mathbf{\theta}^T\mathbf{x} + \epsilon$ with two dimensional inputs and one dimensional output, the analytical solutions of the posterior is a mutilvariate Gaussian $p(\theta|\mathbf{x}, y) \sim \mathcal{N}(\mu, \Sigma)$. Suppose we use a factorized Gaussian as an approximate posterior $q(\theta)=\prod_i q(\theta_i)$ with $q(\theta_1) = \mathcal{N}(\mu_1, \lambda_1^{-1})$, we can show the variance of the approximate posterior factorized Gaussian varies by alpha: $\lambda_1 = \rho_\alpha \Sigma_{11}$ where $\rho_\alpha = \frac{1}{2\alpha}\left[ (2\alpha -1) + \sqrt{ 1- \frac{4 \alpha(1-\alpha)\Sigma^2_{12}}{\Sigma_{11} \Sigma_{22}}}\right]$ is non-decreasing in $\alpha$.

W6 Prior misspecification in ML objectives

The prior can still be misspecified in NN parameters of $p_\varphi(\mathbf{z}|C)$ and the family of distributions we choose for the approximation $p$. For more detailed explanation of limitations on the approximation, please refer to Section 3 likelihood approximation in [1].

[1] Volpp, Michael, et al. "Bayesian Context Aggregation for Neural Processes." ICLR. 2021.

W10 Performance improvements in Table 1.

Please refer to the comment we left for Reviewer 6B6t (Prior misspecification motivation and experiment validation, senario 1) regarding the bad parameterization in the neural networks.

Q1 Misleading gradients.

The second term in Eq (13 (b)) did consider the gradient wrt to $q_\varphi(\mathbf{z}|\mathbb{C})$ which adjust the prior to be closer to the posterior as well.

Q2 A.3 Well specification.

The logic is still to couple the parameters between the posterior and prior model which approximates the ground truth prior. However, RNP dampens the effect of the bad approximation so that the model focuses more on improving the likelihood.

Other issues.

W1. We have shortened the relevant contribution in the introduction section.

W3. We have added the clarification: ``This translate to NPs as the approximate prior model $q_\varphi(\mathbf{z}|X_C, Y_C)$ in Eq 3 can not recover the ground truth prior $p(\mathbf{z}|X_C, Y_C)$ for any parameterization of $\varphi$ ''.

W4. We have revised the proposition 3.2 accordingly.

W5. We have added the reparameterization trick for computing the SGD over the parameters of the posterior.

W7. We have removed the generalization of TNP-A with Diracs in the section.

W8. We have removed the redundant ``tabular regression'' term.

W9. We have added the ODEs for the Lotka-Volterra datset and elaborated on the parameter setting of the dynamics and NP setting for training.

W11. Please refer to the comment we left for Reviewer eR5B regarding MC efficiency.

W12. We have increased the legend size and the figure size. We kept the consistent color and added the legend for red dot indicator.

W13. We have fixed the typos and confusing wordings.

W14. We have moved the notation table to the start of the appendix.

MC efficiency.

We would like to clarify to the reviewer that our RNP framework consumes the same resource as the standard NPs since they usually require multiple samples to better estimate the likelihood model as well. Our overhead computational cost comes from non-analytical solutions for the divergence, whereas our ablations in Fig 5 showed that it takes as few as 8 samples during training to gain better estimates, and one can choose to increase the number of samples during inference for better predictive performance or to reduce it for scalability and real-time applications. We also reported the wall clock time between standard NPs and RNPs in Table 6 (Supplementary) and no significant differences were found.

Hyperparameter sensitivity and automatic tuning.

Please refer to the hyperparameter sensitivity comment for Reviewer T3rx.

Interpretation with robust divergence.

Could the reviewer clarify further about the interprebtability aspect of the model? As far as we understand, neither neural processes nor general neural networks focus on effective interpretability, and RNPs already facilitate better decision makings by providing both the data uncertainty estimation $p(y|\mathbf{x}, C)$ and functional uncertainty estimation $p(\mathbf{z}|{C})$. Regarding relevant healthcare and finance applications, please check the related work [3] where NPs were validated on covid case forecasts and stock pricing prediction (Fig 4 and Fig 8 in [3]).

[3] Wang, Xuesong, et al. "Global convolutional neural processes." 2021 IEEE International Conference on Data Mining (ICDM). IEEE, 2021.

Question 1 RNP efficiency.

One can trade-off between performance and efficiency by increasing or reducing the number of samples in MC for training and inference. In cases where posterior distributions are not standard Gaussians, our framework is equally efficient as the standard NPs.

Dear Reviewer ,

We greatly appreciate your time and effort in reviewing our work.

Regarding the reviewer's initial concern regarding hyperparameter sensitivity and automatic tuning, please check the heuristic (3) we left for the Reviewer T3rx:

To start with standard KL objective ($\alpha \rightarrow$ 1) and gradually decrease $\alpha$ (with granularity according to computational constraints) to 0. The intuition is inspired by KL annealing for VAE models, which starts with a strong prior penalization ($\alpha$ close to 1) to reduce the posterior variance quickly and gradually reduces the prior penalization ($\alpha$ close to 0) and focuses more on model expressiveness.

We have just added the experimental results in Supplementary table 5 to support this strategy, which is more efficient and saves the computational cost of hyperparameter search.

Since the author-reviewer discussion deadline is fast approaching, we kindly ask for feedback on our responses.

We would also like to confirm whether our responses have adequately addressed your initial concerns. If so, we kindly ask whether you might reconsider your original score.

Best regards,

The Authors

Limited novelty.

We would like to clarify to the reviewer that our work goes well beyond using a different divergence for variational inference in neural processes (NPs). Firstly, we are the first to identify that the parameter coupling between the prior conditional and the approximate posterior inherent to NPs amounts to prior misspecification. Secondly, we introduce a new objective that addresses this misspecification and that unifies the variational inference (VI) and maximum likelihood estimation (MLE) objectives. Finally, besides developing the supporting theory in section 3, we show that our RNP method can be applied to several SOTA NP families without changing the models, and provides significant generalization performance improvements over competing approaches.

Hyperparameter sensitivity.

We would like to emphasize that we have carried out a sensitivity analysis on the hyperparameter $\alpha$ in the main paper. The main conclusion across all datasets and models is that $\alpha=0.7$ provides significant performance improvements over the competing approaches. Hence, we recommend this value as default to practitioners and researchers. (See also the additional clarification in section 7.1.) One may also use the prior knowledge to select the $\alpha$ based on the understanding of the misspecification. Nevertheless, we recognize that in other scenarios such as very different datasets and/or models, this default value may not work as well as in our experiments. For this purpose, we have found that cross-validation is an effective tool for finding near-optimal values. Since cross-validation is computationally expensive, we suggest the following heuristics:

1. Start with a value close to 1, which corresponds to standard KL minimization.

2. Only consider $0 < \alpha < 1$ since otherwise we may violate the conditions for the divergence between two Gaussians.

3. Gradually decrease $\alpha$ (with granularity according to computational constraints) to 0. The intuition is inspired by KL annealing for VAE models [1], which starts with a strong prior penalization ($\alpha$ close to 1) to reduce the posterior variance quickly and gradually reduces the prior penalization ($\alpha$ close to 0) and focuses more on model expressiveness.

   [1] Bowman, Samuel, et al. "Generating Sentences from a Continuous Space." Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. 2016.

LLM application.

One can formulate the problem of prompt design for vision-language models as maximizing a contextual probability $\mathbb{C}: \max_{\mathbb{C}} p_{\mathbb{C}}(y| \mathbf{x}, \mathbb{C})$ (Eq (3) in [2]) where $\mathbf{x}$ is an image, $y$ is its corresponding class, e.g., "dog", and $\mathbb{C} \in \mathbb{R}^{n_c \times d_c}$ is a sequence of prompt embeddings to improve the predictive likelihood, e.g., the embedding of "a photo of an [Object]". Similar to our maximum likelihood formulation in Eq (8), it can be viewed as an NP problem, and the parameters $\theta$ of in Eq(8) becomes the context set $\mathbb{C}$ itself. Therefore, the training can facilitate our Renyi objective in Eq (12) to make better inference.

[2] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models." International Journal of Computer Vision 130.9 (2022): 2337-2348.

We thank the reviewer for the prompt response.

Prompt optimization.

Although different LLMs/VLMs have specific architectures, our goal is to learn the prompt embedding $\mathbf{C} \in \mathbf{R}^{nc \times dc}$ which does not depend on the architectures and therefore can be utilized in a general formulation. Recall in the reference [2] the authors also highlighted that ``CoOP models a prompt’s context words with learnable vectors while the entire pre-trained parameters are kept fixed''. In terms of introducing the latent code $\mathbf{z}$, please refer to our eq (11),(12) and the reparameterization trick introduced in the paper:

$\mathbf{z}_k = s(\mathbf{C}, \epsilon_k, \varphi), \epsilon_k \sim \mathcal{N}(\mathbf{0}, I)$

where $s$ is a neural network parameterized by $\varphi$ that takes the context embedding $\mathbf{C}$ and a random noise $\epsilon_k$ as inputs, where both the context embedding $\mathbf{C}$ and the $\varphi$ are learnable parameters. The intuition behind it is instead of modeling the deterministic prompt embedding, we can model its distribution.

Stochastic process consistency.

In the example we presented where the target set is assumed as a subset of the context set, it is consistent under the same parameterization of $\phi$ and $\varphi$. We show the proof with the unified $\varphi$ and $\phi$ here:

$\mathcal{L}(\phi, \theta) = -\mathbb{E}_{q(\mathbf{z}| X_T, Y_T, C, \phi)} \log p(Y_T|X_T, \mathbf{z}, \theta) + {KL}\left(q(\mathbf{z}| X_T, Y_T, C, \phi) \mid q(\mathbf{z}|C, \phi)\right)$

$= -\mathbb{E}_{q(\mathbf{z}| C, \phi)} \log p(Y_T|X_T, \mathbf{z}, \theta) + {KL}\left(q(\mathbf{z}|C, \phi) \mid q(\mathbf{z}|C, \phi)\right)$.

The KL term becomes 0 because {$X_T, Y_T, \mathbf{C}$} = $\mathbf{C}$ and hence, the marginal is consistent with the marginalization of the joint $p(Y_T|X_T, C) = \int p(Y_T|\mathbf{z}, X_T) q(z|{C, \phi}) d\mathbf{z} = \int p(Y_T, Y_C|X_T, {C})dY_C = \int p(Y_T| \mathbf{z}, X_T)p(Y_C|\mathbf{z}, X_C) q(\mathbf{z}| X_T, Y_T, \mathbf{C}, \phi)d\mathbf{z}d{Y_C} = \int p(Y_T| \mathbf{z}, X_T)p(Y_C|\mathbf{z}, X_C) q(\mathbf{z}| \mathbf{C}, \phi)d\mathbf{z}d{Y_C}$.

Hopefully our responses have helped address your concerns. We are more than happy to provide further discussions if needed.

We greatly appreciate the reviewer’s valuable feedback, which has been very helpful throughout this process. We have revisited the initial comments of the reviewer and we genuinely hope that our further clarifications can address some of your concerns.

1. Regarding the reviewer's initial concern for the automatic tuning of $\alpha$, we have previously provided a heuristics. Here, we provide the experimental evidence by comparing with the standard NP objectives. Our heuristics still outperform the baselines across multiple datasets and methods.

2. Relationship with robust divergences.

We have elaborated the relationships with respect to robust divergences in the related work section. We have discussed certain limitations of the robust divergences, e.g., the specifications of the convex function f and deriving its dual form for f-divergence VIs [1]. In terms of their relationships with our work, "As far as we understand, we are the first to analyse the limitations of NPs from the perspective of prior misspecifications, and facilitate robust divergence to enable better NP learning."

[1] Wan, Neng, Dapeng Li, and Naira Hovakimyan. "F-divergence variational inference." Advances in neural information processing systems 33 (2020): 17370-17379.

We are eager to ensure that we have adequately addressed your concerns and are prepared to offer further clarifications or address any additional questions you may have. We look forward to hearing your feedback.