Fisher meets Feynman: score-based variational inference with a product of experts

Thank you for your time in reading our paper. We are happy to hear that you enjoyed reading this paper and appreciated the novelty of the Feynman identity approach and the tractability it adds to this PoE framework.

We respond to your comments and feedback below.

1. Fixed experts and flexibility

“The proposed algorithm only learns the expert weights (\alpha ), while the set of expert distributions (t-distributions) must be predefined. The parameters of these experts remain fixed during training, which could limit the model’s ability to approximate complex target distributions.”

There are many types of approaches to forming a flexible density estimator. One way is the “basis set” view, where you attempt to learn the coefficients of a basis expansion. We deliberately adopt a view that is reminiscent of the basis-expansion perspective, where we begin in a large, expressive set of experts that is then reweighted by minimizing the score divergence. Indeed, Figure 1 (leftmost and rightmost panels) shows that with just two fixed experts, changing the weights can result in very different shaped densities. Furthermore, we show in our experiments that the variational approximation can approximate a variety of target distributions, including challenging geometries such as banana, funnel, cross, and multimodal target distributions (see also new results added, see our response to Review TPhU).

Finally, we note that placing experts randomly and reweighting is one of the simplest first approaches to optimizing this family; there are other more sophisticated approaches that we are actively investigating for future work (see our response to Review xP2U).

2. Feynman identity and broader applicability

“The Feynman identity-based reformulation uses the fact that the density of t-distribution is proportional to the inverse of a quadratic from. This property is essential for applying The Feynman identity, and may not generalize easily to other distribution families. This might limit the broader applicability of the proposed approach.”

We note that the Feynman representation of the product of t-experts leads to an already fairly rich variational family: we can model distributions with different tail weights, and we note that the experts themselves don’t need to be normalized. It is an interesting question what other family of functions remain “closed” under convex combinations, are easy to integrate, and lead to an expressive family of densities. We chose to focus on the t-distribution case given its use in the literature [1], and were able to show in Result 2.1 that this already produces an expressive variational family.

[1] Welling et al. Learning sparse topographic representations with products of student-t distributions. NeurIPS, 2002.

We appreciate your feedback and engagement with the paper. We are glad to hear you found the paper clearly written and highly original. Thanks also for highlighting our theoretical analysis of the algorithm – due to the changing objective function, we could not apply traditional SGD theory for strongly convex functions, and needed to develop a specialized analysis of the algorithm.

Below, we answer your questions and comment on feedback in the review.

1. Performance beyond skewed / heavy-tailed posteriors

“Can the authors elaborate on the benefits (and if any, drawbacks) of the proposed PoE based BBVI technique beyond the skewed and heavy-tailed distributions presented in Section 4? Namely, what is the performance of PoE-BBVI in other posteriors? Is it better/comparable/worse than alternatives?”

First, we would like to emphasize that we evaluated more than just skewed and heavy-tailed posteriors – these posteriors were emphasized because some standard baselines do not work well in these settings (e.g., Gaussians cannot model skew or heavy tails, and the standard flow-based models are not designed for heavy tails).

In our experiments, not only did we study skew and heavy tails, we also studied examples that are light tailed (e.g., garch, funnel, 8-schools, gp-regr) and have complex geometries (e.g., the funnel, the cross, and 8-schools). Furthermore, we found that our PoE method achieved the best resulting Fisher divergence on the posteriordb targets in Figure 4.

That said, we appreciate your curiosity in understanding the performance in a broader class of models. We have added a few other examples that showcase different target distributions. The summary of target distributions is below, where we have indicated which ones are new:

1. Cross shape
2. Funnel
3. Skewed and heavy-tailed
4. Banana / Rosenbrock (new)
5. Multimodal, well-separated modes (new)
6. Multimodal, modes from mixing distributions (new)
7. PosteriorDB - garch11: heavy-tailed + skewed
8. PosteriorDB - 8schools: funnel
9. PosteriorDB - gp-regr: light tailed and symmetric
10. PosteriorDB - sesame: (new)

We believe this provides a diverse set of target distributions that often arise in hierarchical modeling settings. For the final revision, we plan to include additional posteriorDB models.

Additional results

Rosenbrock function (banana)

Multimodal example (well-separated modes)

Multimodal example (modes from mixing distributions)

PosteriorDB: sesame

2. Computational cost and accuracy trade-off of the VI algorithm

“Can the authors elaborate on the computational cost/complexity of computing the PoE within the BBVI framework? what is the tradeoff between compute time and posterior accuracy? can these metrics be reported for the evaluations in Section 4?”

Having more experts will result in an improved posterior accuracy, as we show in the appendix in Figure D.1, but it will also increase the computational cost of solving for the weights. Once the experts are fixed, determining the PoE parameters $\alpha$ comes at a relatively small cost, since we need only solve a quadratic program to determine $\alpha$, which can be done in a matter of seconds, even in high dimensions.

The timings of solving a QP with linear inequality constraints are below for different matrix dimensions $K$ (note that in this work, we only solve QPs up to size $K=100$):

The computational cost of the constrained least squares solve is $O(K^3)$; we note that this cost is not prohibitive for $K < 1000$, and for the problems in this paper, we did not require even this many randomly placed experts. For very large $K$, the cost will become prohibitive, and this provides motivation to work on better methods for expert selection in higher dimensions.

In addition, the cost of computing scores is linear in the dimensionality of the latent variable $z$, and in many real-world problems, evaluating the unnormalizing target and its score is the dominating bottleneck.

Finally, we are also happy to report additional metrics for the computational cost. We will revise this section of the paper to discuss the computational cost in more depth.

3. Expert placement: sensitivity, number of experts, and cost

“Even though it is listed as a limitation, more details on the initialization procedure would be appreciated: How sensitive are the presented results to a good initialization of the fixed collection of experts selected upfront? What is the number of experts used in the experiments, and how do results change with respect to varying it? What's the computational complexity of such initialization procedure?”

Sensitivity: We place experts randomly on a hypercube around each mode, where the bounds are set according to the scale of the target. Thus, if the bounds of the hypercube are set incorrectly, then the tails of the target will not be modeled as accurately. We will add a discussion of this point to the paper.

Varying the number of experts: In Appendix F.2, we describe the number of experts used (including the final number of “active” experts).

How to do results change with the number of experts: We provide an empirical study in Appendix D.2. We show in Figure 2 the KL and Fisher divergences computed for K=10, 25, 50, 75, and 100. Indeed, the divergences decrease as we add more experts. In addition, we report the number of active experts in each case.

Computational cost: With regard to the selection of experts, there is a tradeoff: if the experts are placed randomly, then we may require many, but the cost of each added expert is very small; on the other hand, with very carefully selected experts, we may only require a small number, but we will require a great deal more computation to place them effectively.

We have explored the first regime in which the expert placement is extremely simple and cheap. The main bottleneck of the VI procedure is in the estimation of the weights, not in the expert procedure.

We will revise the current manuscript to make this point about the cost of expert placement more explicit.

I am happy to increase my score, as the provided rebuttal is informative and clarifies many of my raised concerns.

I encourage the authors to make good use of the extra page to incorporate some of these insights, and to clearly point to the appendix for new or existing experiments that I had previously missed (sorry!)

Thanks for your constructive feedback on the paper. We are happy to hear that you found the paper interesting, clearly written, and that you appreciated the overall approach. We respond to your comments and questions below.

1. Expert selection heuristics

“One weakness is the method for selection of experts. This is essentially a heuristic method, that requires hill climbing and hand selecting means, inverse scale, and numbers of experts. Without a more automated and optimizable method for selecting experts and estimating their parameters, is is not clear how broadly useful this method can be, especially in anything but low dimensions of the data. Nonetheless, this work provides a good first step towards exploring this family of models.”

With regard to the selection of experts, there is a tradeoff: if the experts are placed randomly, then we may require very many, but the cost of each added expert is very small; on the other hand, with very carefully selected experts, we may only require a small number, but we will require a great deal more computation to place them effectively.

We have explored the first regime in which the expert placement is extremely simple and cheap, but as suggested by the reviewers, we are also looking at other places on this spectrum, where we consider more sophisticated and expensive ways to choose/adapt a smaller pool of experts (see #3 below for some strategies).

2. Requested additional details in main text

“Could you move more information for both the heuristics for placing experts, and the details of the experiments into the main body? these two aspects seem under described in the text, and the paper would benefit from more details in the main body.”

Yes, due to space, we moved many results and details to the experiments. With the extra page for the final revision, we will be able to better describe the expert placing and move more details of the experiments into the main body (e.g., details currently in Appendix F).

In the main paper, we will revise the experiments section to provide a more complete overview of what to check out in the appendix, so the reader knows where to look, along with a summary of the additional results in the appendix, e.g.,:

• empirical studies on expert placement and varying the number of experts,
• understanding sensitivity to varying learning hyperparameters,
• empirical verification of convergence theory, and
• simulation studies on the normalizing constant estimation procedure.

3. Approaches beyond random expert placement

“Do you have a possible path to move away from heuristics for selecting experts, and instead optimize their parameters?”

Yes, there are many interesting directions that we are pursuing but did not make it into the paper for lack of space or being out of scope; here we chose to focus on other elements of the paper (e.g., studying the PoE family and theoretically characterizing convergence) and to use the simplest approach for expert selection. Below, we describe three approaches of increasing complexity.

One idea is to use a score-based metric for finding where to place experts (in contrast to what we are currently doing, which involves placing a lot of experts randomly and pruning): that is, we check where the scores between the target and variational approximation are a poor fit, and place new experts to patch up the scores.

Another approach is to select experts in a greedy “boosting” way (e.g., [1]) – that is, we iteratively repeat: 1) adding a new expert according to regions where there is large score mismatch, and then 2) running the described reweighting procedure to refit the weights of the experts.

A final approach is that this family can be directly optimized using gradient descent. In addition, this family could be used in an amortized inference setting by letting the expert parameters and weights be a function of a shared set of parameters (over a data set). While we expect such approaches to utilize more compute, it could be interesting in settings where the user has a large computational budget for training, and would like posterior inference to be fast at test time.

We are actively investigating all of these directions.

[1] Miller et al. Variational boosting: iteratively refining posterior approximations. ICML, 2017.

4. Bias from empirical loss estimate

“Do you a have a notion of how big of a bias is introduced by the empirical estimate for the loss? Do you have any experiments that show if this significantly affects the optimization?”

The bias introduced due to sampling from $q(z| \alpha^{(t)})$ instead of $q(z| \alpha)$ does not prevent the iterates from converging to the optimal parameters. In particular, we prove in Theorem 3.1 that the iterates $\alpha^{(t)}$ obtained from solving Equation 27 converge to the optimal parameters (under no misspecification; otherwise, they converge to a neighborhood of the optimal parameters). We will add a sentence in this section clarifying this point.

Finally, we also show experiments investigating the convergence under different values of $\eta_t,$ which controls the step size, and we find that the optimization is not very sensitive to this parameter.

Thank you for your time and your feedback on our paper. We are glad that you appreciated the “elegant foundations” and novelty of the Feynman representation of the product of experts, the clarity of the writing, and the theoretical grounding of this work, such as the auxiliary variable formulation and the analysis of the algorithm’s convergence.

Below, we respond to your feedback and questions on the paper.

1. Expressiveness of the variational family

“Though this work motivates the development of this novel variational approximation from the perspective of having expressive and practical approximate posteriors, I do not see strong supporting evidence in the work to conclude either of these.”

We show that the family is expressive in several ways. First, the Feynman representation shows that the product of experts can be represented by a continuous location-scale mixture of t-distributions; indeed with just two experts, Figure 1 shows a variety of behaviors can be captured. Second, we show how the expressivity of the family impacts the convergence of the algorithm: in particular, the iterates converge to a neighborhood of the optimal weighting, where the radius of the neighborhood is determined by the amount of misspecification of the variational family (due to e.g., expert selection). Finally, we show how the PoE family can be used to model a variety of target distributions – we have added several new benchmarks in the rebuttal (see discussion below in #2).

2. Empirical results on posterior benchmarks

“A limited evaluation has been conducted here and without more exhaustive testing on practical applications and uncertainty estimation benchmarks, it's hard to ascertain the quality of the posterior inferred.”

Thanks for this feedback. We have now run the procedure on several new target distributions, including standard synthetic benchmarks and real-world data benchmarks from posteriordb. We believe that this addition adds to the diversity of targets studied in the empirical results. In particular, we have results covering the following targets:

1. Cross shape
2. Funnel
3. Skewed and heavy-tailed
4. Banana / Rosenbrock (new)
5. Multimodal, well-separated modes (new)
6. Multimodal, modes from mixing (new)
7. PosteriorDB - garch11: heavy-tailed + skewed
8. PosteriorDB - 8schools: funnel
9. PosteriorDB - gp-regr: light tailed and symmetric
10. PosteriorDB - sesame: (new)

We believe this provides a range of target distributions that often arise in hierarchical modeling settings. The initial results for the new targets can be found in our response to Reviewer TPhU, and we plan to include additional posteriorDB models in the final version.

3. Expert placement at random locations

“The proposed approximation has very elegant foundations but, in the authors own words, optimising the parameters is complicated and without a strategy to place the experts at the right spots this approach cannot infer the right expert parameters. In this regard I would also like the authors to comment on the case where experts are placed at completely random locations. This seemingly non-negotiable requirement to correctly initialise the experts, shrinks the applicability of the proposed variational approximation.”

We understand that you are concerned about the need to place experts at the “right” locations. But it is not the case that there is “ground truth” expert placement and the model fails unless it recovers that ground truth.

A concrete example is when the target is itself a product of t-experts – our expert placing procedure has no prior knowledge of the true locations, nor does it try to “infer” these locations. Instead, we place down a lot of experts and use the flexibility of reweighting to match the scores. We show that our procedure can still recover the target distribution without knowing or finding the true locations.

4. Relationship with implicit VI approaches

“According to Result 2.1 text in l 113- l 115 , this PoE can be viewed as a continuous mixture of t-distributions. In this regard I would like the authors to comment on relevant approaches in Implicit variational inference [1][2] where continuous mixtures of Gaussians have been proposed as expressive variational approximations. How does the proposed approach compare to such formulations?”

The implicit VI setting is an interesting one, and thanks for pointing us to these references. We will add a discussion of this setting and include additional relevant references to the updated paper.

Via the Feynman representation and Result 2.1, the product of experts we consider can be viewed from the framework of implicit VI with a different kernel (t-distribution) and mixing measure (Dirichlet). In particular, the product of experts has a particular way of combining the auxiliary variables via Equations 15 and 9–10, whereas the typical implicit VI setting specifies a neural network with a mixing distribution over the parameters of that neural network.

In this special case of continuously parameterized models, we are able to identify tools that lead to both an expressive family of densities but also a tractable normalizing constant via the Feyman representation. Our goal in this work has been to understand the potential and limitations of this special case.

5. Comparison with mixtures of Gaussians

“What happens when you initialise the mixture components of a GMM at the right spots and only optimise the mixing proportion parameter? In terms of divergences, does the GMM not capture the target distribution accurately? How much better would a PoE be compared to this? When a GMM with a large number of components is initialised for the same target, does standard VI not infer a sparser GMM by driving down mixing proportion values for some of the components?”

The weights in a mixture of experts play a different role than a product of experts. A useful way to distinguish between the two constructions is that mixture weights act like an “or,” whereas product weights act like an “and”. While the latter may seem restrictive, we also don’t require the experts themselves to be normalized, unlike in the mixtures case.

Thus, we expect the two models to excel in different types of applications. To give an extreme and pedagogical example, if the target distribution is a student’s t distribution, we need an infinite mixture of Gaussians to model this.

In addition, we note there are differences in optimization: products of experts are more amenable to score matching, as the objective is linear in the weights, leading to a constrained, nonnegative least squares (NNLS) problem. On the other hand, mixtures do not have efficient solutions for score-based divergences (or even the KL divergence), and are often optimized with gradient descent.

These differences are important because the NNLS objective used in the PoE setting tends to learn sparse solutions, so the parameters are driven down to 0. It’s unclear whether optimizing the GMM weights (e.g., using "standard VI" or ELBO optimization) would lead to a sparse solution.

We will add a discussion to the revised paper highlighting the conceptual differences between these models.

5. Computational cost of using many experts

“From Figure D.1 it seems that you have to always start with a very high number of experts, off which some are found to be redundant. Does this not lead to an increase in computational cost? Is there any way to be more efficient with this initialisation?”

Using more experts increases the cost of determining the PoE parameter alpha, but this comes at a relatively small cost, since finding alpha boils down to solving a quadratic program, which can be done in a matter of seconds, even in high dimensions.

We performed timings for solving a QP with linear inequality constraints for different matrix sizes $K$ (note that in this work, we only solve QPs up to size $K=100$):

However, your question about other ways of initializing is an interesting one. We believe there are many ways this can be done, and for this paper we decided to focus on the simplest strategy. For future work, we have been actively investigating other strategies. See our response to Review xP2U for some concrete examples.

6. Additional clarity points

“l 25 >Each of such gradient evaluation..... . I think this line is a little vague. In my opinion, it really depends on the downstream task or purpose which determines whether a sample or a gradient is more useful. Kindly consider rephrasing this line or adding more information.”

Thanks for this feedback. Due to space limitations, we cut a contextual footnote related to this after the phrase “considerably more than what is provided”.

The footnote said: “Here is a simple illustration of this point: suppose that $p$ is univariate Gaussian. Then one can determine the mean and variance of $p$ from the derivatives of $\log p$ at any two points in $\mathbb{R}^1$, whereas from $n$ samples (but no scores) one can only estimate these moments to $O(1/\sqrt{n})$ accuracy.”

We are happy to add this back to the final revision of the paper.

“It's hard to draw conclusions from the figures in the appendix, namely figure E.1 and F.2. In F.2 the legend is not readable.”

We will increase the sizes of the figures and the axis fonts.

I thank the authors for posting a detailed reply and individually addressing all my concerns. Please refer to the bullet points below for my response.

• 1 and 2, thanks for adding more experiments. I am sure that these would go on to help readers who wish to develop approaches in this area or apply this one. Although with my comment about expressivity, I was also hinting at testing some higher-dimensional cases. Kindly help me with what is generally the dimensionality of the posterior you have tested your approach with.

• 3, I am not fully convinced since this is not a solution rather a workaround that could in future have limitations. It's good to see that the cost of using a large number of experts is low. Thanks for adding that result.

• 4, thanks, glad that this could improve your work. And thanks for explaining.

• 5, there are two 5's above, so I am referring to the one about comparisons with GMMs.

  It’s unclear whether optimising the GMM weights (e.g., using "standard VI" or ELBO optimisation) would lead to a sparse solution.

  I think you should investigate this, as it is quite relevant and the explanation is a bit vague. I am sure that there are sparse priors for GMMs that could enforce the same effect. I think this could improve your work.

• 6 Thanks for accepting my suggestion.

Overall, I thank the authors for their effort. I am not 100% convinced by the evaluation, but they have improved their manuscript, and I maintain that this work is original and interesting.