Hybrid Kernel Stein Variational Gradient Descent

We thank the reviewer for their time in providing their review and feedback.

Weaknesses:

1. We acknowledge that some proofs are presented as straightforward extensions and one-liners. However, we would like to mention some key technical details in the Appendix. Lemma B.1 is non-trivial and is required to establish the optimality of the h-SVGD update direction. Some oversights in proofs in the existing literature have been addressed. For example, our Assumption (B2) replaces a previous very restrictive assumption which, through Lemma B.2, makes Proposition 4.7 a stronger result than its counterpart in Korba et al. (2020). Also, Remark 3 addresses the interchangeability of expectation and inner products, which has not been addressed in the SVGD literature to our knowledge.

2. Diversity of kernel choice: As noted, the developed theory does not require kernels $k_1$ and $k_2$ to be related, but the simulation results were based on kernels of the form $k_2=f(d)k_1$. We have now included additional BNN experiments with a range different kernels in Appendix D. Please see Main Response for further details.

3. The current work serves to establish the theory of h-SVGD for the first time, and to demonstrate its ability to alleviate variance collapse. However, we agree that choosing the scaling function is a worthwhile direction for future research, and we now explicitly mention this in Section 5.2 and the conclusion.

4. While, as correctly noted, h-KSD is not a valid discrepancy measure, the descent lemma (Theorem 4.2) shows that the KL divergence bounds can be written in terms of the vanilla KSD and guarantees a decrease in the KL divergence at every step. Some additional discussion has been added to Section 4.3. See the Main Response for further discussion.

Questions:

1. The set within which the optimisation takes place is chosen as a subset of this intersection. For standard kernel choices, Remark 2 ensures that this intersection will be either $\mathcal H_1$ or $\mathcal H_2$. This amounts to optimising within a unit ball on $\mathcal H_1$ or $\mathcal H_2$, although with a norm induced by the sum Hilbert space.

2. To our knowledge, the SVGD literature does not contain kernel choices of this type.

3. If $k_2$ is the RBF kernel, a larger bandwidth allows a steeper gradient $\nabla k_2$ when evaluated on distant particles. For intuition, think of the plot of $\exp(-x^2/h)$, so that a higher bandwidth increases the repulsive force, and thereby mitigates variance collapse. We now provide a comment on this in Appendix D.

4. The conditional dependence structure refers to probabilistic graphical models that factorise into a set of lower dimensional inference problems. In this example, variance collapse is alleviated by working in lower dimensions. We have slightly modified the beginning of Section 5 to make this more clear.

5. One possible reason is that the Yacht dataset has the smallest number of records (306) and the second smallest number of features (6) of the ten datasets considered.

6. The paper has been updated with the definition of $\Phi_p^{k_1,k_2}$ now just before equation (9).

We thank the reviewer for their time in providing their review and feedback.

Weaknesses:

1. Significance of h-SVGD: Please see our Main Response for full details. We have now improved our presentation of our empirical results in Section 5 to more clearly demonstrate that h-SVGD has similar performance to SVGD in terms of test RMSE and test log-likelihood, but improved (or equivalent) variance estimation. This is now most clearly seen in Figure 1, revised Figure 2, and new Figure 5 in Appendix D, which summarises performance over 9 kernel variants.

2. Evaluation of kernel families: Thank you for this welcome suggestion. We have now added results using additional different kernel families (9 variants in total) for the repulsive kernel to Appendix D, including the kernel adopted in D'Angelo et al. (2021). The theory still applies to this latter kernel choice; in particular, the conditions for the descent lemma (Theorem 4.2) are satisfied, as detailed in Appendix D.

3. Log likelihood not correlated with dimension averaged marginal variance: To our understanding, log-likelihood is a measure of test accuracy. Better variance representation doesn't necessarily mean that log-likelihood is improved.

4. Better intuition: We have now included some additional commentary in Section 4 to provide more intuition on the results. In particular, we note Remark 1 and Remark 2 (which has now been moved slightly earlier in the paper) for some intuition on the main result (Theorem 4.1). Further discussion has also been added around Theorem 4.2 on how the KL divergence will always decrease under h-SVGD.

Questions:

1. D'Angelo et al. (2021)'s kernel: As requested we have now included simulations using this kernel for the repulsive term, as outlined above.

We thank the reviewer for their time in providing their review and feedback.

Weaknesses:

1. Strong experimental results for h-SVGD: Please see our Main Response for full details. However, note that the mitigation of variance collapse in high dimensions is demonstrated in Section 5.1 and Figure 1 (a-c), which shows a significant improvement in estimating the true variance in high dimensions over SVGD, even when the number of dimensions is greater than the number of particles ($\gamma>1$). Also, revised Figure 2 now more clearly demonstrates an increase in DAMV for the BNN example, with consistent performance in test RMS and log-likelihood. Please also see new Figure 5 in Appendix D for additional qualitatively similar results with other kernel choices (9 kernel variations). Overall, the general performance of h-SVGD is comparable with SVGD but with the advantage that h-SVGD has superior performance in mitigating variance collapse.

2. While the primary aim of this paper is to provide theoretical justification of h-SVGD, we recognise that it is useful to support this by also demonstrating the performance of h-SVGD in areas that are directly related to this theoretical development; namely performance with regard to different families of kernels. This has now been done, and has been implemented in Appendix D. We feel that comparison to other competitor methods (such as GSVGD and S-SVGD) is somewhat adjacent to the focus of this paper, and is more relevant for research that makes algorithmic contributions in this field.

Questions:

1. In terms of computational costs: the cost of updating the particles for both SVGD and h-SVGD is $O(N^2)$ at each step. We have added this statement at the beginning of Section 5. The SVGD update can be slightly faster when $k(x,y)$ can be reused to compute $\nabla k(x,y)$ (this applies to the RBF kernel for example), but the order is still $O(N^2)$.

We thank the reviewer for their time in providing their review and feedback.

Weaknesses:

1. Significance of h-SVGD: Please see our Main Response for full details. We have now improved our presentation of our empirical results in Section 5 to more clearly demonstrate that h-SVGD has similar performance to SVGD in terms of test RMSE and test log-likelihood, but improved (or equivalent) variance estimation. This is now most clearly seen in Figure 1, revised Figure 2, and new Figure 5 in Appendix D, which summarises performance over 9 kernel variants. In addition, at the start of Section 5 we now comment that other previous methods (such as GSVGD and S-SVGD) also tackle variance collapse, but they do so at a far greater computational cost. In particular, S-SVGD requires additional computation of the optimal test directions and GSVGD requires additional computation at each step to update the projectors. The advantage of h-SVGD is that there is no added computational cost over regular SVGD.

2. Convergence properties of h-SVGD: The question is about whether h-SVGD can minimise the KL divergence effectively: see our Main Response for full details. As noted, the descent magnitude is h-KSD (Theorem 4.1), which is not a proper discrepancy. However, the descent lemma (Theorem 4.2) bounds the decrease in KL divergence in terms of a proper discrepancy, namely the KSD of one of the kernels. We have now included further details in Section 4.3 that show for proper choice of step size, the KL divergence is strictly decreasing at all times, meaning that the h-SVGD algorithm avoids cases where $\mathbb S_{k_1,k_2}(\mu_\ell^\infty,\nu_p)=0$ but $\mu_\ell^\infty$ and $\nu_p$ are not equal almost everywhere.

Questions:

1. Remark 2 mentions that either $\mathcal H_1 \subseteq \mathcal H_2$ or $\mathcal H_2 \subseteq \mathcal H_1$ for many common choices of kernel (e.g. RBF, IMQ, log-inverse, or Matérn). This includes when $k_1$ and $k_2$ are from different families mentioned above. So the intersection $\mathcal H_1 \cap \mathcal H_2$ will be either $\mathcal H_1$ or $\mathcal H_2$. We are not aware of applications in the SVGD literature that use kernels other than those listed above. Remark 2 has been reworded slightly and moved so that it appears just after Theorem 4.1.

2. As mentioned in response to the weaknesses above, the advantage of h-SVGD is that it alleviates the variance collapse at no additional computational cost, whereas S-SVGD and GSVGD require additional computations of the optimal test direction or the projectors at each step.