Understanding the difficulties of posterior predictive estimation

We thank the reviewer for their detailed comments. We will address the typos, equation formatting, and other minor suggestions as is and offer comments to other concerns below.

... Bayesian CLT approximations ... provide additional detail ...

While there are several versions of Bernstein-von Mises, we believe Theorem 10.1 from Vaart (1998) suffices for our use case. Note that Theorem 10.1 states the result in terms of the true parameters. For the extension to the maximum likelihood estimate, see the discussion between Lemma 10.3 and Lemma 10.4.

In short, we assume that the posterior $p(z \vert \mathcal D) \approx \mathcal N(z_{\textrm{MLE}}, \frac{1}{|\mathcal{D}|} I^{-1}(z_{\textrm{MLE}})),$ where $I$ is the estimate of the Fisher information matrix evaluated at the maximum likelihood estimate; such that, $I^{-1} (z_{\textrm{MLE}}) = -(\frac{1}{|\mathcal D|} \nabla^2_z \log p(\mathcal D \vert z))^{-1}$. Since the size of the dataset cancels out, we get the expressions in equations 65, 66, and 67. In the revision, we will add the discussion on Bernstein-von Mises and the above clarification after Corollary D.2.

[1] Vaart, A. W. van der. "Bayes Procedures." Asymptotic Statistics. Cambridge: Cambridge University Press, 1998.

Assumptions on the data-generating process for ... equation 70.

In spirit, this assumption is not that different from assuming the data-generating distributions are the same. The quantity in eq. 70 ($\frac{1}{|\mathcal D|}S_{\mathcal D}^{-1}$) is the estimate of the Fisher information matrix (FIM) under different datasets. We essentially assume that the train and test data sets are similar enough such that estimates (MLE and FIM) under these datasets will be the same. As the reviewer notices, it is tempting to only assume that the train and test data distributions are similar. However, even after assuming the same distribution, we will require additional assumptions for a result in terms of estimates under two given datasets. This assumption about the estimates is precisely the reason we (very explicitly) keep the result informal. We will add this discussion after eq. 70, and if the reviewers suggest, we will reword the assumption in terms of FIM.

What is being defined in definition C.2?

There is no specific assumption on the form of the posterior. We are essentially writing Bayes' rule. However, we define $V$ explicitly as we use it as a function of data $\mathcal D$ in later expressions. We made an organizational choice to aid the reader, but we can move it to an inline definition if preferred.

The discussion of approximate inference ... but didn't provide insight ...

The main insight is that for approximate posteriors that closely resemble the true posterior, we expect the relationships from Section 2 to hold as is. However, for arbitrary distributions, it is hard to make a precise statement beyond "SNR depends on how much the posterior $q_\mathcal{D}(z|\mathcal{D}^∗)$ varies from the prior $q_\mathcal{D}(z)$." Intuitively, the training data only enters these equations through the conditioning, or, equivalently, from the approximate posterior $q_\mathcal D(z)$. Since arbitrary approximate posteriors may have arbitrary relationships with training data, it is not immediately obvious how to make a more precise statements. We hope that future research can explore this further. We will add this discussion at the end of Section 3 and in the limitations paragraph in Section 6.

What is the main (practical) insight I am supposed to take from section 2.2?

The analysis in Section 2.2 is complementary to the main claims of the paper. From the earlier parts in Section 2, we know that when the training and test datasets match and are large enough, the posteriors in eq. 3 are similar, and the SNR is high. The same is implied by the two conditions at the end of section 2.2. When $\xi_\mathcal D$ is large, the training dataset will be large (note that $\xi_\mathcal D$ includes both the statistics and the number of data points, see eq. 10). When the points $\xi_\mathcal D$ and $\xi_{\mathcal D + \mathcal D^*}$ lie close to the ray emanating from the origin, the datasets have to be similar. Combined together, these two conditions also support the view that when the datasets are large and similar, we expect the SNR to be high. We favored the illustration in Figure 6 in place of words to derive the main takeaway. However, we will add some of this explicit discussion to the main text.

Overall, we thank the reviewer for these insightful comments. We strongly believe that the concerns raised by them will make the paper stronger. We look forward to addressing any remaining concerns. Otherwise, we feel confident in our abilities to add the requested explanatory text, and hope the reviewer will consider a strong recommendation for our work.

Thank you very much for your detailed and encouraging review. We appreciate your recognition of our theoretical analysis and the empirical validation of our proposed LIS method, as well as your insights into potential computational challenges. Your constructive feedback is truly invaluable.

Thank you so much for your thoughtful and constructive review. We appreciate your recognition of the theoretical contributions and the thoroughness of our analysis. Your feedback is invaluable and greatly encouraging.

For the camera-ready version, we will plan to use the extra space to add more commentary on the takeaways from sections 2.2 and 3 (as pointed out in response to the reviewer hMnG). We will also use the extra space to move more of the related works discussion to the main text (as indicated in response to the reviewer CNiq). In fact, we are open to moving the model details to the appendix and, instead, moving all of the related works section into the main text. Please let us know if you have a strong preference for one or the other.

We thank the reviewer for their feedback. We will address the minor concerns and offer some specific comments below.

Why is SNR a good metric for analyzing the quality of Monte Carlo estimation of the PPD?

We study SNR because it is equivalent to relative variance and bakes in the idea of how large the estimator variance is relative to the target quantity. The idea of relative variance becomes crucial when the target quantity is numerically small, as in the case of $\textrm{PPD}_q$ values. To make this precise, let’s consider an example where the $\log \textrm{PPD}_q= -100$ (for reference, all estimates of $\log \textrm{PPD}_q$ in Tables 1, 2, 3, and 4 are lower than $-100$). Also, consider an unbiased estimator $R$ with variance $\mathbb V(R)=10^{-20}$. In the absolute sense, the variance of this estimator is low; however, $R$ carries more noise than signal. To see why, note that $\textrm{PPD}_q=\exp(-100) \approx 3.72\times10^{-44}.$ Intuitively, $R$ varies on the scale of $10^{-10}$ (standard deviation) and will produce noisy approximations of the target value that is the order of $10^{-44}$. SNR naturally captures this intuition: $\textrm{SNR}(R) = \textrm{PPD}_q/\sqrt{\mathbb V [R]} \approx 3.72\times10^{-34}$ and flags the estimator as poor.

Several works before us have studied the SNR of unbiased estimators of numerically small quantities (as cited in lines 59-61). For the works cited in the main paper, the focus is to study the behavior of gradient estimators since the value of the gradients can become very small as the optimization proceeds. In our extended search, we found two more works that study the SNR of gradient estimators for VI [1, 2] and one more work that studies SNR for the policy gradients in RL [3] (the definition of the SNR in [3] is different from our ratio of moments definition; however, the idea to study the SNR is similarly motivated). In hindsight, we agree this discussion should be more prominent. We will add these citations and the above discussion to the already existing discussion in lines 59-61.

[1] Liévin, Valentin, et al. "Optimal variance control of the score-function gradient estimator for importance-weighted bounds." NeurIPS, 2020.

[2] Rudner, Tim GJ, et al. "On signal-to-noise ratio issues in variational inference for deep Gaussian processes." ICML, 2021.

[3] Roberts, John, and Russ Tedrake. "Signal-to-noise ratio analysis of policy gradient algorithms." NeurIPS, 2008.

How do the theoretical results here influence the way that practitioners estimate PPDs or the way they select approximate posterior distributions?

We believe our theoretical results are of independent interest and also provide useful insights for practitioners. First, our work serves as a strong cautionary note. We hope that practitioners monitor (and report) the SNR value of their PPD estimates alongside the $\log \textrm{PPD}$ value. Second, whenever the SNR values of the naive estimator are low, our work shows that this does not necessarily reflect on the accuracy of the approximate posterior but simply on our ability to accurately estimate PPD. In cases of low SNR, we suggest practitioners use approaches like LIS to improve the reliability of the estimates and obtain a clearer picture of the relative performance of the different approximate posterior methods.

We briefly touch on the above discussion in the conclusion paragraph in Section 6 and will expand on this in the update.

 Related work is contained only in the appendix, whereas model details are placed in the main paper after the conclusion.

Due to space limitations, we moved the related work section to the appendix. In the camera-ready version, one more page is allowed, and we plan to use it to bring more of the discussion on the related works into the main paper (note that a small paragraph is present in Section 6). If the reviewers believe that the paper will benefit from moving all of the related work section to the main paper, then we propose moving the Model Details section to the appendix to create more space.

 The related work section goes back only a decade or so, which makes it difficult to assess the significance of the results in this paper.

Any research review can fall short of being exhaustive. We believe we made an honest attempt to cover the relevant literature. We cite research on PPD evaluation in Section B, and we will further expand on our motivation to study SNR (as mentioned in this rebuttal). If we are missing some relevant work, we will be more than happy to include it. However, we respectfully disagree that it is difficult to judge our work. We will be obliged if the reviewer can point us to specific literature that we have missed, and without which it becomes difficult to assess the significance of our analysis.

Overall, we thank the reviewer for their time and hope that they will reconsider their position in light of the explanations and other reviews.