Variational Learning of Fractional Posteriors

some claims … insufficiently supported by the empirical

Lines 12 (right) and 87 (right) are general remarks on robustness of fractional posteriors, citing others. Statement on line 44 (right) follows these, but may be misunderstood. We will change to “an alternative to”.

Alignment with prior is substantiated by Table 2 (last col), Fig 4&5, and Fig 7 & Table 5.

Experimental

only two-component mixtures

Beyond 2 components, the marginal likelihood landscape is complex (line 328, left, citation therein). The main text uses 2 components for clarity and focus. Beyond this, we need more exposition to separate the confounding effects due to the complex landscape. Nonetheless, C.1.2 gives results for 4 components.

linear regressions … lack sufficient theoretical justification

Indeed, it’s more justified to regress against $\ell^{-2}$ (C.1.1). This gives better interval lengths, but worse coverages. Other strategies are possible (last sent. in sec. 6).

Comments (numbered)

1. Yes. See caption (also Fig 4 of Kingma&Welling, 2022]. marginal-CDF$\in [0.1, 0.9]$
2. All results are based on the variational approximation optimised on train set. The empirical error-bars are for 10 runs, a different seed each to initialise the NN.

With “ELBO/div”, we know that the fractional posteriors are closer to the prior for smaller $\gamma$ in the same sense of KL. This can't be seen from “Objective/div” because Renyi-divergence differs with $\gamma$. See also line 358 (left)

3. Ok
4. We haven't systematically investigated. Table 2 hints at stability from the relatively small error bars (3 x stddev) by using 100 samples per datum and 10 runs (details in C.3).

Questions (numbered)

1. Sensitivity analysis of $\gamma$ must be done within posterior families (sec 2.2). Current results show the quality of the bound and the closeness of the approx. fractional posterior to the prior.

2. We will acknowledge this experimental limitation in an additional section.

Current 2d latent space allows posterior illustrations (Fig 4 to 6). We use NNs from [Ruthotto & Haber, 2021], with only 88,837 parameters for 2d latent space, so results can be obtained on free platforms (sec C.5). We need richer NNs for more complex data.

We are also careful and use the correct likelihood: continuous Bernoulli. Otherwise, bounds in Table 2 are not convincing.

We expect similar benefits compared with existing objectives such as ELBO or $\beta$-VAE’s, when the goal is either better evidence bounds, or more stable computation of fractional posteriors, or both.

3. Degeneracy isn't necessary for optimality; but it isn't avoided and is also a solution, as stated in A.4. There, Eqs 3&4 say if we explicitly enforce $q(u)\neq0$ at multiple locations, optimality is still possible. So, we can design $q(u)$ to be non-degenerate if we wish. Further work can be on crafting training dynamics towards non-degeneracy.

4. For FashionMNIST, using 4d latent space and the following $\gamma$s for our primary bound:

• 1 (ELBO): train/test bound=1220/1200; FID=58.3
• $10^{-3}$: train/test bound=1231/1219; FID=56.8
• $10^{-5}$: train/test bound=1231/1219; FID=55.8

So for 4d, bound for smaller $\gamma$ gives better results. Overall 4d results better than 2d (see C.4 for existing values), as expected for a richer model with more parameters.

We haven't analysed the MC approx error.

5. We don't directly address posterior collapse. If we must, we might seem to be encouraging collapse, but it's more complicated and demands more discussion. Figure 4(d) gives a prelude, where the fractional posteriors as a whole aggregate towards the prior, but posterior for every data point is different.

6. With respect to image generation from prior (sec 5.3), we learn a decoder that operates well with the prior, which can be simple. At the same time and within the same objective, the NN are optimised with respect to the fractional posteriors that are close to the prior.

Post learning the empirical prior requires its model to be sufficiently flexible to approx the posteriors, and a separate procedure to learn its parameters.

So we believe our proposal is simpler.

Which is more practical depends on context within a wider ML/AI sys — e.g., if posteriors exist from previous intensive training/tuning, the empirical prior approach might be preferred.

7. The samples are noisy also because NN architectures for the encoder & decoder are simple with 89K parameters for 2d latent space. For CIFAR10, we add one CNN layer (total 3) each for encoder & decoder, total 530K parameters for 32d latent space. Number of train/eval samples reduced to16/128; epochs reduced to 300 (C.3 gives current settings). Results using SageMaker StudioLab:

• $\gamma=1$ (ELBO): train/test bound=1229/1172; FID=141
• $\gamma=10^{-5}$: train/test bound=1238/1184; FID=135

Our bound for $10^{-5}$-posterior gives better results. Better results need more complex architectures, e.g., ResNet-1001 with 10.2M parameters.

Relation To Broader Scientific Literature

Thank you. We will try our best to incorporate these into the paper.

Other Strengths And Weaknesses

The method involves nested integrations and Monte Carlo estimates (especially in the hierarchical and semi-implicit formulations), which may lead to high computational overhead. This can become particularly problematic in high-dimensional latent spaces or when dealing with complex models.

As in our reply to Reviewer H3En, the triple integral in section 2.3.2 can be reduced to a double integral (see section A.3 and last row of Table 3). The remaining double integrals are necessary since we have a hierarchical construction, involving $\boldsymbol{u}$ and then $\boldsymbol{z}|\boldsymbol{u}$. Fortunately, depending on the application, one may not need such a construction if we set $\gamma$ to be small so that the approximating family can be simple, as discussed in section 2.2.

Questions For Authors

We have done preliminary work on such extensions, but it is not trivial. We leave such matters for future work to keep the current paper focused. Subject to space constraints, we may suggest approaches in relation to other frameworks in an additional future work or discussion section, especially on pitfalls to avoid based on our experience.

Methods And Evaluation Criteria

For the methodology, only the derivation of $\mathcal{L}^{bh}$ is not clear to me. As in A.3, there are two possible objectives, but the main text uses the one with both $\boldsymbol{u}$ and $\boldsymbol{u}$’ in the integrand. I do not see why this one is chosen.

This one is chosen because we have its experimental results at time of submission. We now have experimental results for the alternate and simpler bound in A.3 — the empirical results are very similar (for example, the test objective for $\gamma=0.5$ is $1608.0\pm25.4$), and the implementation is simpler and does not involve triple integrals. We also have proof showing non-degeneracy of the implicit distributions is not neccessary, similar to A.4.2.

We will replace the currently chosen bound with the simpler bound in the main text. The current bound will be placed in the appendix. The essential arguments in the experimental section remain the same.

For a theoretical paper like this, I think the benchmark datasets are enough to support the claims. However, it would be much better if the comparison with $\beta$-VAE is also demonstrated in the experiments.

The $\beta$-VAE is demonstrated in C.4 for generating images from the prior, and this is referenced from the main text in section 6 (line 409, right column). In addition to the parameters for $\beta$-VAE currently in C.4, we have additionally tried with $\beta$ taking values 5 and $10^2$, giving FIDs 78.4 and 99.1. The conclusions are the same. We will include these additional results in the final version.

Subject to space constraints, we will move some of these results to the main text in the camera-ready version.

Since $\beta$-VAE for fractional posteriors is provably less tight than ELBO, we do not include the results in section 5.2/Table 2.

Theoretical Claims

Section 3.1 contains three cases where components in the objective could be derived analytically. Given that modern Bayesian models can be written as probabilistic programs with program tracing and autodiffs, I suppose most of the derivations could be automated in practice.

Yes indeed, for the most part. In addition, we believe knowing the derivations and derived formulae have pedagogical value, and can give rise to new algorithms and/or more efficient updates in future work. For example, autodiff will not be able to choose $1/\gamma = 1 + 1/n$ (section 3.1.2); nor is it able to decide to apply $\mathcal{L}_\gamma$ first on $q(\boldsymbol{u})$ and then ELBO for $p(\boldsymbol{c})$ (section 3.1.3), and in this order. After these are determined, and perhaps with further simplification of the bounds and equations, then autodiffs can be applied readily.

Essential References Not Discussed:

We will mention [1]-[3] and relate them to our work.

Questions For Authors:

Answered above.

We suspect misunderstandings. We ask for further clarifications below.

Claims & Evidence

contributions ... confounded

Our key contribution is a lower bound that also approximates fractional posteriors. Having both at once is new in ML and statistics. Moreover, ELBO is a special case. So, we fill a gap between standard VI (ELBO) and fractional Bayesian inference.

Fractional posteriors have been shown to be more robust (sec 1 2nd para), and have applications in calibration (sec 5.1). A lower bound allows principled maximisation of the hyperparameters (ML-II), common in Bayesian ML (sec 5.2). Having both at once benefits generation from VAE decoder (sec 5.3).

We bring the bound in sec 2 to expts in sec 5 by developing complex posterior constructions (sec 2.3), parameter updates (sec 3) and MC estimates (sec 4).

$L_{ELBO}^{\gamma}$

$L_{ELBO}^{\gamma}$ essentially changes the likelihood of the model, so it is an ELBO to that changed model, and may not lower bound the original model (depends whether $p(D|z)>1$). In contrast, ours is a theoretical lower bound to the original model. It holds for all $\gamma\in(0,1)$ and all $q$.

Nonetheless, the optimal solution of $L_{ELBO}^{\gamma}$ indeed approximates the fractional posterior of the original model. It's the same as $\beta$-VAE’s objective, by dividing by $\gamma$ (Table 3). If we only want a fractional posterior, then this suffices. If, in addition, we wish to estimate the hyperparameters of the model, as is common in ML, then our lower bound is effective (Table 2).

statistically better in any sense than naively using the ELBO ...?

We show our bound to be computationally more stable and to give better results than $\beta$-VAE for the FashionMNIST in C.4 (referenced in sec 6). We have not analysed the properties such as convergence rates. Please clarify if we misunderstood the question.

computational benefit?

No. Indeed, KL is mathematically more convenient. Computationally, if implemented using sampling (e.g., for VAE), we swap the log and sum operations, include multiplicative and additive constants, and use more than one sample (line 807).

Works that propose new divergences …what is uniquely new or desirable about the proposed divergence

We provide a variational objective that lower bounds evidence and enables estimation of approx. fractional posteriors. This is not fulfilled by $L_{ELBO}^{\gamma}$ (nor $\beta$-VAE).

Methods & Evaluation

... a surrogate for some divergence, but which one? ... equivalent to this divergence up to a constant ... a strict surrogate?

Optimality is at the fractional posterior so it's not a divergence to/from the Bayes posterior. We do not think it’s equivalent to any known divergence up to a constant (the evidence). We can define a divergence (from the fractional posterior) based on our bound: $D(q||(p(z),p(D|z),\gamma)) = L_{evd} - L_\gamma$, where the triple defines the target posterior.

We can better answer if “strict surrogate” is defined.

$\gamma$ is a specification of the model, not the inference

$\gamma$ is not a specification of the model. From a Bayesian viewpoint, the model is fully specified by likelihood $p(D|z)$ and prior $p(z)$. Given a data set, this fixes the exact marginal likelihood or data-evidence $p(D)$ (MacKay, 2003).

One may compute $p(D)$ and $p(z|D)$ using sampling, such as MCMC. For complex models in ML, variational methods are developed (see 1st para of paper). We propose a new variational method that leads to approximate fractional posteriors. Our method has a parameter $\gamma$ for the inference, but it does not change the model.

We admit there are schools other than Bayesian. In particular, in the regularisation community, the end objective is treated as the model, like the reviewer alludes to. To put our paper in the correct setting, it begins with “Exact Bayesian inference is ...”.

Optimal in what sense?

In the sense of giving a tighter bound to the evidence (1st sent in para). We will reiterate at the end of para.

modifying the model … statistically sound.

We emphasise that the model is fixed (given hyperparameters) and evidence is fixed (given data) in the Bayesian view. Our lower bound to the evidence is mathematically sound. It changes neither $p(D|z)$ nor $p(z)$; max wrt $q$ also changes neither. But max wrt $p(D|z)$ or $p(z)$ changes the model.

Experiments

Our evaluation is sound: In sec 5.1, the model is fixed and the bounds are comparable for relative tightness to the same exact evidence. In sec 5.2, the model changes because we optimise the decoder, changing $p(D|z)$ (right para starting line 318). Here, the bounds are comparable for the quality of optimised models within the same family (i.e., same NN architecture) on the same data.

We should have used the word “tighter” more carefully. We will update the abstract; 4th para in intro; last para in sec 2.2; intro to sec 5. 4th and 5th para in sec 5.2; and conclusion.