Convolutional Deep Kernel Machines

Please see Overall summary at the top, which describes major changes, including dramatically improved performance on CIFAR-10 (from 91.7% in the original manuscript to 92.7% in this version), and a revamped introduction section that more clearly explains the motivation of our work.

Theoretical contributions of the convolutional DKM. Absolutely! We have rewritten the introduction to highlight that there is a large body of work in the NNGP and NTK literature, attempting to push up the performance of these methods. There is good theoretical reason for doing this. In particular, we want to know whether infinite-width neural networks capture the "essence" of what makes neural networks work so well, or whether there is something missing. Indeed, it does seem that NNGPs underperform relative to finite NNs, suggesting that NNGPs are indeed missing something fundamental. One hypothesis is that this missing piece is representation learning. However, we can't really be sure that this is the case, given the numerous practical differences between NNGPs and finite NNs. One way to confirm this hypothesis would be to develop an infinite-width NNGP-like approach that somehow includes representation learning. We do exactly that, and find that we do indeed get improved performance over NNGPs (92.7% vs the best NNGP result of 91.2% from Adlam et al. 2023). Furthermore, the DKM framework provides a handle (the regularisation strength hyperparameter $\nu$) to smoothly interpolate between the NNGP kernel regime and the fully-representation-learning regime (see Table 1).

Questions:

The authors state that DKMs are derived from deep GPs where the width of the intermediate layers goes to infinity, but it seems that the kernel function in Section 4 is calculated based on finite number of channels (i.e., is finite). Can authors provide some clarification?

Here, we're taking inspiration from neural networks to derive an inter-domain inducing point scheme for the infinite width limit, so we necessarily intermix the kernel and feature-based viewpoint. There's two tricks here as we move from the feature-based to the kernel-based viewpoint.

First, we use expressions like Eq. 15, which are valid when $\mathbf{H}$ has a finite or infinite number of features.

Second, in e.g.\ Eq. 19 and 20, we use a standard trick from the NNGP literature. In particular, the distribution over all features (indexed $\lambda$) is IID. Therefore, we consider a particular feature, (the $\lambda$th feature) in Eq. 19 and 20, but the derivation applies to all (potentially infinitely many) features.

It is not clear whether the kernel matrix is positive-definite by defining using learnable parameter , or at least a priori (i.e., before training). Can authors give some intuition on this?

The kernel defined in (Eq. 17) is just the covariance of a random vector, so it has to be positive semi-definite. Specifically, Eq. 17 can be rewritten as,

$\mathbf{K}^{\rm{Conv}} = \mathbb{E}[\mathbf{f}_{\lambda} \mathbf{f}\_{\lambda}^T]$

where, $\mathbf{f}^T = (\mathbf{f}^{\rm i}\_\lambda \mathbf{f}^{\rm t}_\lambda)^T$. We use $C$ to define $\mathbf{f}^{\rm_i}$, but the kernel would be positive semi-definite for any choice of $\mathbf{f}^{\rm_i}$.

Why is the convolutional DKM compared with other models only on CIFAR-10 but not CIFAR-100 in Table 3?

Previous methods take so long to run that they tend to only give results on CIFAR-10 (but we're happy to include comparisons on CIFAR-100 if you know of any relevant ones).

Please see Overall summary at the top, which describes major changes, including dramatically improved performance on CIFAR-10 (from 91.7% in the original manuscript to 92.7% in this version), and a revamped introduction section that more clearly explains the motivation of our work.

Benefits of our approach

At present, there are three primary benefits of our approach.

First, our work addresses a number of theoretical questions arising in the NNGP literature. These include:

• Is it possible to obtain an infinite-width, NNGP-like limit in which representation learning can occur?
• Current NNGPs underperform relative to finite NNs. One hypothesis that might explain this underperformance is that NNGPs have a fixed kernel, and thus lack representation learning. While that hypothesis is plausible, it is not easy to confirm, given the huge differences between NNGPs and finite NNs. To confirm this hypothesis, we really need a variant of the NNGP that somehow includes representation learning. That's precisely what we develop in this paper. And we show that by introducing representation learning into NNGPs, you indeed get a considerable performance boost (from 91.2% in Adlam et al. (2023) to 92.7% here). Furthermore, as seen in the updated results in Table 1, varying our regularisation strength allows us to smoothly interpolate between the NNGP regime and the representation-learning regime. All this suggests that it is indeed the lack of representation learning that is causing at least part of the underperformance of NNGPs relative to finite NNs. This has been hypothesised in the past (e.g. David MacKay. Introduction to gaussian processes. 1998) but we are the first to empirically show this on meaningful datasets like CIFAR-10.

Second, our work forms a bridge, allowing us us to connect the fantastic collection of prior work that you point out in your review to current work on NNGPs.

Third, improved performance against previous kernel methods. Our best result is 92.7% as opposed to previous results from kernel methods with parameters (89.8%; Mairal 2016) and without parameters (91.2%; Adlam et al. 2023).

Additional related work Thanks for highlighting this awesome collection of related work! We have updated the Related Work to include a thorough discussion of all of it (also see below). We hope that our paper can drive renewed interest and attention towards this collection of prior work, by highlighting:

• the potential excellent performance of deep kernel methods on datasets such as CIFAR-10
• theoretical relationships between the broad family of deep kernel methods and "fashionable" topics, such as representation learning in infinite-width neural networks. In any case, we'd be excited to follow up with the authors of this work! (Anonymity permitting, of course.)

At a high-level, there are two key differences to this past work.

First, this work lacks a theoretical link to representation learning in infinite-width neural networks, and thus cannot be used to address the theoretical questions addressed by our work. But there are other differences in the details.

Second, it seems that a lot of this work (e.g. Mairal et al. 2014, Mairal 2016, Johan 2017, Dexiong et al. 2019, Achten et al. 2023) uses a (convolutional/recurrent) neural network like structure to obtain a finite-dimensional projection/approximation of an underlying fixed kernel. In that sense, this work resembles (or could be used to get) an efficient approximation of NNGP-like kernels. In contrast, our objective (Eq. 5) is fundamentally not a projection/approximation of an underlying, fixed kernel. Instead, it allows "full" flexibility in the Gram matrices/kernels, resembling exactly the type of flexibility in the kernel that you'd get in a deep GP or a finite, deep neural network.

[1] Bohn, B., Rieger, C., & Griebel, M. (2019). A representer theorem for deep kernel learning. The Journal of Machine Learning Research, 20(1), 2302-2333.

Deep kernel learning is of course very different to our approach. In deep kernel learning, you train a neural network, and feed the outputs of that network into a kernel. In contrast, in a DKM approach, there is no neural network: there are no features and no weights. Instead, there are just kernels (or structured approximations to kernels).

Tonin, Francesco, et al. "Deep Kernel Principal Component Analysis for Multi-level Feature Learning." arXiv preprint arXiv:2302.11220 (2023),

This is an unsupervised feature extraction method.

Title too broad? We agree that the term "deep kernel machine" is perhaps quite broad. Your comment raises a bit of a dilemma: the use of the term "Deep Kernel Machine" in this context was introduced by Yang et al. While their paper was formally published in 2023 at ICML, the paper (and terminology) was established on arXiv as early as 2021. Moreover, there are other terms such as "Deep kernel learning" which seem very broad, but in practice are used to refer to a very specific way of combining a neural network and a kernel, ("Deep kernel learning" Wilson et al. 2015). Do you have any suggestions about how to resolve this dilemma?

Uncertainty The three key advantages of our approach are outlined at the top of this response. Unfortunately, as we take the infinite-width limit, we lose the straightforward Bayesian interpretation, so the implied uncertainty properties are unclear, and we leave this for future work.

Minor aspects:

Some typos and non-mathemcatical edits, e.g., the 2nd paragraph under Table 3 on page 8.

Thanks, fixed!

In the provided codes, there are different variants of nonlinear kernels. But I didn't notice a clear descriptions on how these nonlinear units are chosen or considered.

In practice, we just use the arccos kernel corresponding to the relu nonlinearity, to ensure a close correspondence to CNNs.

How about the empirical efficiency difference on varying the inducing points setups in Table 2?

We have now added this to the paper in Table 2. We include the times below for the reviewers' convenience.

In Table 1, many factors are investigated, and in different sections mostly the performances did not changge drastically in the compared techniques and were similar to other sections. The evaluation is comprehensive, but the conlusion is less focused, as we would hope to quickly get which factors are the most important ones. I these results don't change much and all attain good results, and then a mild suggestion or a global defaulting setting can be a take-away message.

See the "Benchmark performance" paragraph for a description of how these experiments informed our choice of defaults for subsequent experiments.

Please see Overall summary at the top, which describes major changes, including dramatically improved performance on CIFAR-10 (from 91.7% in the original manuscript to 92.7% in this version), and a revamped introduction section that more clearly explains the motivation of our work.

Eq. 18 Agreed: this isn't intuitive, but it is one of the key contributions in our framework.
We could use inducing points that look like images/feature maps, but if we did that the memory usage would be far too high (as explained in the paper). Instead, we worked hard to come up with memory-efficient inter-domain inducing points that are just a flat vector, with no spatial structure. You can see that because the $F_{i,\lambda}^{\rm i}$ and $H^{\rm i}_{i', \mu}$ just have channel indices ($\lambda$ and $\mu$) or inducing point indices ($i$ or $i'$), and don't have any spatial indicies.

But in Eq. 18, we have the convolutional weights, $W_{d \mu, \lambda}$, which does have spatial structure; how does that work? The answer is that the ``inner'' multiplication by C,

$\sum_{i'} C_{di , i'} H^{\rm i}_{i', \mu}$

has a result that has indices $i, d, \mu$. i.e.\ this inner multiplication gives a bunch of image patches, which we can combine with the weights, $W_{d \mu, \lambda}$! In short, we multiply spatial-less inducing points by various weights C to form spatial patches, and then convolve these with the convolutional weights W.

Benefits of Convolutional DKMs

Experiments only show accuracy and NLL, and while this is fine for standard machine learning models, the benefit of the Bayesian approach is mostly in the other benefits it adds besides plain accuracy. It would make the paper much better if you show that your VI approximation still holds these properties with additional evaluations on these benchmarks like calibration or out-of-distribution detection.

The two key advantages of the DKM approach are described in the "Overall summary". First, the DKM approach gives improved performance over previous kernel methods. Our best result is 92.7% as opposed to previous results from kernel methods with parameters (89.8%; Mairal 2016) and without parameters (91.2%; Adlam et al. 2023). Second, our results have important implications for our theoretical understanding about the importance of representation learning in NNs (see modified Introduction). In particular, our new results show much more clearly in Table 1 how our regularisation strength hyperparameter allows the DKM framework to smoothly interpolate between the fixed NNGP kernel regime and the fully-expressive representation learning regime. Unfortunately, as we take the infinite-width limit, we lose the straightforward Bayesian interpretation, so the implied uncertainty properties are unclear, and we leave this for future work.

Writing We have copy edited the paper with a native English speaker.

Please see Overall summary at the top, which describes major changes, including dramatically improved performance on CIFAR-10 (from 91.7% in the original manuscript to 92.7% in this version), and a revamped introduction section that more clearly explains the motivation of our work.

Novelty of convolutional inter-domain inducing point method for DKMs As discussed in the Introduction and Related Work, standard inducing point approaches for Convolutional GPs don't work, because they require features at each layer, while the DKM only gives us Gram matrices. Working out how to get things to work efficiently, while working in the Gram matrix domain is highly non-trivial, and the key reason that the math in the paper is a bit painful.

Additionally, the dramatic differences between the methods come out in the experiments. Our radically different approach from convolutional DGPs allows us to scale to much deeper models (e.g. Dutordoir et al. 2020 only went up to three layers), and hence allowed achieve dramatically improved performance (Dutordoir et al. only got 76.17% on CIFAR-10, while we got 92.7%).

Derivation of DKM objective Please see Appendix A (which was present in the original manuscript) for a first-principles derivation, or the original DKM paper (Yang et al. 2023).

Appropriate baselines You're right that the best comparison perhaps isn't the "parameter-free" NNGP methods. The best comparison would be against other kernel methods that also have trainable parameters. We didn't originally include them, as there aren't many such methods, and they don't tend to perform as well as NNGPs. Nonetheless, we've updated Table 3 to include such methods, including convolution DGPs (up to 76.17% on CIFAR-10 with Dutordoir et al. 2020), and convolutional kernel networks (up to 89.8% with Mairal, 2016). Our approach gives considerable performance improvements, with our best result being 92.7%.

The benefits of convolutional DKMs The two key advantages of the DKM approach are described in the "Overall summary". First, the DKM approach gives improved performance over previous kernel methods. Our best result is 92.7% as opposed to previous results from kernel methods with parameters (89.8%; Mairal 2016) and without parameters (91.2%; Adlam et al. 2023). Second, our results have important theoretical implications for our theoretical understanding about the importance of representation learning in NNs (see modified Introduction). In particular, our new results show much more clearly in Table 1 how our regularisation strength hyperparameter allows the DKM framework to smoothly interpolate between the fixed NNGP kernel regime and the fully-expressive representation learning regime, thus demonstrating that representation learning is a key component for successful deep models.

Unfortunately, as we take the infinite-width limit, we lose the straightforward Bayesian interpretation, so the implied uncertainty properties are unclear, and we leave this for future work.