Hybrid Kernel Stein Variational Gradient Descent

We greatly thank the reviewer for their time in providing their review and feedback.

Weaknesses:

• We appreciate the suggestion for a new title and have updated the title to "Convergence aspects of hybrid kernel SVGD".
• We would like to draw attention to Section 3.5 where the gradient flow is explored in the more general case where $k_1$ and $k_2$ may be kernels from different families. Conditions are given for the existence of a Wasserstein gradient flow in Proposition 3.7. Also, there is a concrete calculation in Proposition 3.8 demonstrating the non-convergence in the mean field limit for a specific target.
• We would like to highlight that Tables 3 and 4 in Appendix B are comparing h-SVGD against S-SVGD, not vanilla SVGD. S-SVGD is a variant that addresses variance underestimation. Table 3 shows that h-SVGD outperforms S-SVGD at alleviating variance collapse on 9 out of 10 datasets, and runs much faster. Table 4 shows that both h-SVGD and S-SVGD are comparable in terms of RMSE and LL test metrics, but as per Table 2, they both outperform regular SVGD on these metrics. Hopefully this makes it more clear as to which algorithm we are comparing h-SVGD against in tables 3 and 4.
• We appreciate the reviewer's attention to detail in this comment. All uses of SSVGD have now been updated to S-SVGD for consistency.

Questions:

• Non-convergence in the mean field limit is a property of h-SVGD. Note that in the two concrete examples (Corollary 3.4 for $k_2=ck_1$ and Proposition 3.8 for the general case), the mean field limiting distribution has the same mean. Therefore, the introduced bias increases the variance, but does not affect the mean. In this way, h-SVGD samples from a distribution with an overestimated variance, and this offsets the variance underestimation in the finite particle setting.
• The case of two RBF kernels with different bandwidths is explored in Section 3.5. This includes concrete calculations in the proof of Proposition 3.8. The theoretical analysis of two kernels with different forms (e.g. RBF and IMQ) is generally intractable because of the Fourier transforms involved (see Equation 22). Based on Proposition 3.8, we do not expect mean field convergence under other more general settings.
• Since the utility of h-SVGD on image analysis tasks has already been demonstrated (D'Angelo et al., 2021), we chose to focus on other benchmark datasets. We chose the BNN example because it was studied in the original SVGD paper by (Liu and Wang, 2016) as well as the S-SVGD paper (Gong et al., 2021).
• Capturing variance of the posterior is critical for uncertainty quantification in Bayesian statistics since variance is often used as a measure of confidence in an estimate (we have included this comment in the second paragraph of Section 1). Taking the BNN example, variance collapse in this setting leads to a set of networks with almost identical parameters, which defeats the purpose of taking a Bayesian approach.

We greatly thank the reviewer for their time in providing their review and feedback.

Weaknesses:

Major weaknesses:

• The reviewer is correct - thankyou. This is a typo and Assumption (A1) has been updated to $\lim_{| x|\to\infty}V(x)=+\infty$.
• Thank-you. Following this suggestion, we have included some interpretation and commentary on the assumptions in Section 3.1. We have also included a reminder on the Wasserstein gradient flow definition in Section 3.4. Finally, we have also added some commentary on the interpretation of Proposition 3.6.

Minor Weaknesses:

• Due to the symmetry assumption on the kernels, Assumption (B1), both options are equivalent. For consistency with equation (2), we have updated equation (6) to $( x,\cdot)$ as suggested.
• Line 198 should remain $\mathcal H_1$ as there is no $\mathcal H$ in this setting, only $\mathcal H_1$ and $\mathcal H_2$.
• Following the correction to the typo in Assumption (A1) mentioned above, the remark in line 246 is now consistent.
• The $p$ in line 250 has been changed to $\pi$, the target distribution. We appreciate the reviewer's attention to detail.
• Following the correction to the typo in Assumption (A1) mentioned above, the normal density now satisfies Assumption (A1).
• Since the functions $K_1$ and $K_2$ both have domain $\mathcal X\subseteq\mathbb R$, by symmetry we mean $K_1( x) = K_1(- x)$. As per the suggestion, we have added this definition in Assumption (B1).

Questions:

• Following the correction to the typo in Assumption (A1) mentioned above, those two assumptions are now independent.
• Assumptions (A2) and (A3) can be interpreted as controls on the decay of the tails of the distribution, ensuring well-behaved tails. We have now included a comment on this in the paragraphs following the assumptions.
• Following the corrections to the typo in Assumption (A1), and the update from $|V( x)|^q$ to $|\nabla V( x)|^q$ in the first part of Assumption (A2), Gaussian potentials and mixtures thereof satisfy all four assumptions. Many other continuous distributions from the exponential family also satisfy these assumptions.
• We present h-SVGD as a sampling method that, despite sampling from a different distribution than the target, alleviates variance collapse in practice. This happens because the distribution from which the particles are sampled overestimates variance in the mean field limit, as seen in Corollary 3.4, and this overestimation offests the variance underestimation that occurs in the finite particle setting with regular SVGD.

We greatly thank the reviewer for their time in providing their review and feedback. We appreciate the acknowledgement of our contributions on the theoretical constraints and practical advantages of hybrid SVGD

We greatly thank the reviewer for their time in providing their review and feedback.

Weaknesses:

• We appreciate the insightful observation that the improvements in RMSE and LL may be due to factors other than mitigation of variance collapse. We have included a comment on this in Section 4.2.
• We would also like to address the comment about limited support. The BNN experiment offers support for improved variance in two ways. Firstly, through improved DAMV on all but two datasets. And secondly, there is a well known decomposition of the MSE, namely that $\rm{MSE} = {\rm var} + {\rm bias}^2$. Since RMSE is improved on all but one dataset, this suggests an improved variance estimation.

Questions:

• We would like to emphasise that Corollary 3.4 is a result in the mean field limit, a common setting throughout the SVGD literature. However, the issue of variance underestimation occurs in the finite particle setting, which is more commonly encountered in practice. One can interpret the increased variance in the limiting state distribution $\rho^*$ as counteracting the variance underestimation in the finite particle regime, thereby leading to more accurate variance estimates in practice. We appreciate the observation and have chosen to include additional commentary on this point in the discussion of results in Section 4.2.