Particle Semi-Implicit Variational Inference

Thank you for your efforts in reviewing our work and for the helpful feedback and suggestions. All typographic errors will be fixed.

Figure 1: The current caption makes it difficult to interpret the figure. It would be helpful to explicitly mention that lighter shades represent smaller μ\muμ values, while darker shades represent larger μ\muμ values.

This is a good suggestion. We shall make the necessary amendments.

Figure 2: The second sentence of the caption is somewhat confusing. Referring to Table 1 for additional clarification might improve the reader's understanding.

Thank you for pointing out this mistake. The comment referred to the notation $\mu_{\pm \sigma}$ where $\mu$ is the average and $\sigma$ is the standard deviation computed from $10$ independent trials. This shall be fixed in future iterations.

Thank you for your hard work on our submission, and for recognizing the value of our work.

How sensitive is PVI to the choice of kernel and number of particles?

We found that PVI the choice of kernel is an important one. In Section 3, we discuss the implications of the kernel choice more explicitly.

As for the number of particles, after a certain number, we found that there were diminishing/no returns. In all our experiments, we used $100$ particles and found that this was sufficient for good performance. We did not finetune this quantity.

Can the theoretical analysis be extended to cover the γ = 0 case used in practice?

Unfortunately, we do not currently know how to do this. Taking $\gamma \rightarrow 0$ will result in a non-Lipschitz drift which would violate the assumptions of our theoretical analysis. The bounds in Appendix F.4 would diverge. In order to extend our analysis, one would need to establish existence and uniqueness via other arguments. However, the theory of McKean-Vlasov SDEs is at this stage much less developed than that for simple SDEs and we are not aware of any existing arguments which apply (if any) for the $\gamma = 0$. Our belief is that this reflects the current state of knowledge about McKean-Vlasov SDEs rather than being a fundamental issue.

Despite claiming in Section 2 that PVI can learn potentially more expressive variational approximations than other SIVI methods, no experiments empirically demonstrate this advantage.

In our experiments, we found that PVI performed well and outperformed all other existing semi-implicit VI methods. The advantage of our expressivity results in obtaining better approximations. In Section 5.2, the quality of the approximation can be seen through lower sliced Wasserstein scores and lower rejection rates; and in Section 5.3, we use MSE as a proxy of posterior quality for which PVI achieves the best overall performance. We also provide additional experiments as part of the rebuttal that compares MCMC samples with PVI samples on a Bayesian Logistic Regression example studied in [1, Section 5.4]. Although this does experiment does not demonstrate an advantage, it is encouraging that our method aligns with MCMC samples.

[1] Yin, Mingzhang, and Mingyuan Zhou. "Semi-implicit variational inference." International conference on machine learning. PMLR, 2018.

Are there specific types of problems where PVI is expected to significantly outperform existing methods?

Whenever it's difficult to specify a variational family which is sufficiently expressive to capture the structure of the posterior even in this semi-implicit setting (e.g. in substantially multimodal settings) we would expect this approach to come into its own. In our numerical examples, we have focussed on fairly comparing the method with alternatives in settings in which they (other semi-implicit algorithms) do perform well (rather than contriving settings in which they fail) and have seen that the particle-based approach is extremely competitive even on the examples used to showcase earlier methods: we felt that demonstrating that nothing is lost in using this more general framework when other methods work provided a strong motivation for using it.

However, we do not have an example where other methods will fail terribly (we have not actively tried to find one as we had focussed on comparing our method with early approaches using examples chosen by those authors to showcase their work; we can certainly explore this further and would welcome suggestions). Due to the mode-seeking behaviour of reverse KL that other methods minimize (at least approximately), we expect that these methods would (at worst) recover one of the modes for well-defined models. Please do share if you have any suggestions on this and we will include it in any future versions of the manuscript.

How does the choice of preconditioner affect PVI's performance?

When a problem/kernel is ill-conditioned, we found that the preconditioner can be used to stabilize the training procedure. In the Bayesian neural network experiment (Section 5.3), we utilise this trick. In situations where the algorithm is already stable, it may not be required and the algorithm performed well regardless.

Clarity in explaining the flexibility of PVI's variational approximation.

Thank you for pointing this out. We shall amend the final paragraph of section 2 to make this distinction clearer and more explicit.

Thank you for your time spent reviewing our work.

what's the meaning of 'Here the ± denotes the average ...'?

Thank you for pointing out this mistake. The comment referred to the notation $\mu_{\pm \sigma}$ where $\mu$ is the average and $\sigma$ is the standard deviation computed from $10$ independent trials. This shall be fixed in future iterations.

Thank you for the effort you spent on our work.

However, the current paper does not provide details of how the SVI, UVI, and SM are implemented.

We perhaps didn't make this clear enough (and can address that in subsequent versions), but line 281 of the main text points out that all hyperparameter choices and runtimes for all methods are provided in Appendix H. We usually follow the recommended settings found for each competing method in their respective papers (except for when we did not find explicit recommendations). Source code for our implementations of all methods was provided at the time of submission.

There is a question of whether runtime is a meaningful measure of fairness given that there can be discrepancies based on how an algorithm is implemented. As the runtimes provided in Appendix H demonstrate, other algorithms were not disadvantaged. For instance, we followed the recommendations of UVI paper which resulted in an algorithm that exceeded our time computation by an order of magnitude. In the supplementary PDF, we include the requested figures for the density estimation task. To achieve high speeds with PVI, one takes advantage of the fact that the particle update is easily parallelizable.

First, none of these simulations directly show the accuracy of the posterior inference for a complete Bayesian model.

We agree. We now provide (preliminary) experiments on the Bayesian Logistic Regressions setup studied in prior works [1, Section 5.4]. In the attached PDF, we provide plots that compare the quality of the posterior against MCMC samples obtained in [1]. It can be seen that PVI obtains a posterior quality that closely matches that of the MCMC samples.

[1] Yin, Mingzhang, and Mingyuan Zhou. "Semi-implicit variational inference." International conference on machine learning. PMLR, 2018.

Does the particle method suffer from the curse of dimension problem? How does the number of particles scale with dimensionality to maintain accuracy? all the simulations are in low-dimension.

As noted in Appendix 5.3, the Bayesian neural network example has dimensionality {331, 81, 101} which one can claim is not "low dimensional". Like other methods, PVI utilizes neural networks to allow the method to scale to high dimensions while retaining the expressivity of the particle method. Experimentally, we found that $100$ particles were sufficient for good performance across the examples considered. We kept this constant throughout our experiments and did not finetune it. The source code for our implementations was provided at the time of submission.

Last, the paper writing is not coherent in some places The non-coercive is not defined in Proposition 2.

The coercivity definition is a standard one for which we provided an explicit reference for it in the main text on line 128. We shall add the definition to the Appendix.

We will reread the manuscript carefully and address any similar issues; we'd be happy to address any specific concerns you may still have.

The paper mentions the non-coercive "is closely related to the problem of posterior collapse"; what is the relationship exactly?

In the amortized VI, one definition of posterior collapse is when the approximate posterior $q(x|z)$ does not depend on $z$ and collapses to the prior. In the context of PVI, one can prove that the functional $\cal{E}$ is not coercive by using the kernel $k(x|z)$ that does not depend on $z$. Since the kernel is $k$ and approximate posterior $q$ often share the same functional form, e.g., they ($k$ and $q$) are often parameterized with $\cal{N}(\mu(z), \sigma^2I)$ where $\mu$ is some neural network, the comment suggests to the reader why it may be unsurprising if they share this issue.

How to compute the precondition matrices in Eq 13 is not discussed

The preconditioning matrix was discussed in Appendix G.2 for which we have provided a reference at line 209 which is within the main text.

Does the regularization in Eq 4 introduce bias in the posterior inference?

Yes. As with any non-parametric estimation problem with a finite sample size, regularization is an important algorithmic step. Loosely speaking, one can decrease the weighting of the regularization as the number of samples goes towards infinity to remain consistent.

Does the theoretical analysis in Sec. 4 provide guidance in designing the algorithm?

Not directly. The primary purpose of the analysis is to provide "theoretical underpinnings give a rigorous basis for the practical algorithm" to quote Rev JYYZ.

How to check coercive and l.s.c. in Proposition 3?

There are various proof techniques that one can use to show coercivity and lsc. Our proposed regularizer satisfies the assumptions of Proposition 3.