Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?

Thanks for your detailed review and your clear recommendations for improvement. We appreciate your positive feedback on our work.

Suggestions on main results presentation and is there any particular reason why you only include NLPD results in the appendix?

Thank you for your suggestions on which results to include in the main paper. Space constraints were the main reasons to place some of the results on the appendix but we agree that including them in the main text will be valuable.

Data sorted by the first dimension: Why not sort by the last dimension instead / what if the data is not sorted at all?) Do you have any specific reason for this? And how does this compare to batches which are simply sampled uniformly at random?

The reason we sorted the data along the first dimension is to follow the experimental setup used by Chang et al. (2023) for the UCI experiments, but any other dimension could have been used. If we were to sample batches uniformly at random, we would expect a behaviour similar to the middle column of Figure 1. After a few initial batches, enough of the input space would be covered causing the number of inducing points to asymptote to a particular value.

All considered datasets [...] seem to be quite small...

We agree with the reviewer that adding larger-scale datasets could improve the paper. The main reason for not including them was that finding suitable large-scale real-world datasets for where GPs are an appropriate model to use is challenging. However, we were considering including a large-scale synthetic dataset in the final version, which would allow us to evaluate the scalability of our method in such scenarios.

Thank you for your detailed and thorough review. We appreciate your perspective on the significance of our work.

[Q] Focus on regression problems.

As you noted, the main reason is the theoretical guarantees and the bound from Titsias (2014), which was derived for GP regression with Gaussian likelihoods. In this case, the variational KL divergence can be made arbitrarily small by increasing the number of inducing points. This no longer holds in classification, where the likelihood is non-Gaussian

[Q] Validity of Eq. (5) without $\Phi$ term

Eq. (5) is no longer a bound on the marginal likelihood of the full dataset, so the theoretical rigour of the original bound is, strictly speaking, lost when omitting the $\Phi$ term. However, it remains a valid lower bound for the new data, given the posterior carried from previous batches. As we add more inducing points, $\Phi$ approaches zero, and in the limit, Eq. (5) recovers the full bound.

This is where our argument becomes empirical: we show that by adding enough inducing points, we can keep $\Phi$ small enough for Eq. (5) to behave as a proper ELBO during training, and that this is sufficient to maintain performance on the full dataset.

[Comment] The last term of Eq. (7) could be ignored from an optimization point of view.

From an optimisation point of view, yes, this is right.

[Q] Must $\mathcal{U}$ vary with $M$ in the same way as $\mathcal{L}$?

We are free to use a different number of inducing points to calculate $\widehat{\mathcal{U}}$ compared to what we use in the lower bound. Using more inducing points for the upper bound leads to a stricter stopping criterion, allowing us to stop adding points to the lower bound sooner. This is useful since only the inducing points used for the lower bound are retained for future batches. So, by increasing computation for the upper bound slightly, we ultimately reduce the number of inducing points retained lowering the overall cost.

[Q] How do $\mathbf{q(a)}$ and $\mathbf{q(b)}$ mix for the third iteration $\mathbf{c}$?

In our algorithm, we keep the old inducing point locations $\mathbf{Z}_o$ fixed and choose the new set of inducing points from among the locations in the new batch $X_n$. Consider three batches $\mathbf{X}_1, \mathbf{X}_2, \mathbf{X}_3$ with inducing variables $\mathbf{a}, \mathbf{b}, \mathbf{c}$. For the first batch, we construct the variational approximation $q(\mathbf{a})$ using inducing points $Z_1 \subset \mathbf{X}_1$. For the second batch, we keep $\mathbf{a}$ and select new inducing points $Z_2' \subset \mathbf{X}_2$, forming $\mathbf{b} = \mathbf{a} \cup f(Z_2')$. The updated variational posterior is:

$$q(\mathbf{b})\propto p(\mathbf{b}) \mathcal{N} ( \hat{\mathbf{y}} ; \mathbf{K}\_{\hat{\mathbf{f}} \mathbf{b}} \mathbf{K}\_{\mathbf{b}\mathbf{b}}^{-1}\mathbf{b}, \Sigma\_{\hat{\mathbf{y}}} )$$

where $\mathbf{K}\_{\hat{\mathbf{f}} \mathbf{b}} = \begin{bmatrix} \mathbf{K}\_{\mathbf{fb}} \\ \mathbf{K}\_{\mathbf{ab}} \end{bmatrix}$ , $\Sigma_{\hat{\mathbf{y}}} = \mathrm{diag}([\sigma_y^2 \mathbf{I},\, \mathbf{D}\_{\mathbf{a}}])$, $\hat{\mathbf{y}} = \begin{bmatrix} \mathbf{y}_n \\ \mathbf{D}\_{\mathbf{a}} \mathbf{S}\_{\mathbf{a}}^{-1} \mathbf{m}\_{\mathbf{a}} \end{bmatrix}$. For the third batch, $q(\mathbf{b})$ now summarises all past information. We select new inducing points $Z_3' \subset \mathbf{X}_3$ and form: $\mathbf{c} = \mathbf{b} \cup f(Z_3')$ which leads to the update:

$$q(\mathbf{c})\propto p(\mathbf{c}) \mathcal{N} ( \hat{\mathbf{y}} ; \mathbf{K}\_{\hat{\mathbf{f}} \mathbf{c}} \mathbf{K}\_{\mathbf{c}\mathbf{c}}^{-1}\mathbf{c}, \Sigma\_{\hat{\mathbf{y}}} )$$

with updated kernel matrices and statistics, using $q(\mathbf{b})$ instead of $q(\mathbf{a})$.

[Q] Is the noise hyperparameter fixed, as stated in the Experiments section?

Thank you for pointing this out. This is a typo: the noise hyperparameter is not fixed. The text should read, "the variational distribution, noise and kernel hyperparameters are optimised [...]". We will correct this.

Figure 2A: Thank you for the suggestion. We will emphasise that $M$ shown at the top is for VIPS using a subscript $M_{vips}$.

Stress testing: Thank you for your suggestions and for clearly indicating that additional experiments were not required at this stage. Please see our response to Reviewer Me8M regarding larger datasets.

Technical challenges in Bui et al. (2017). Does VIPS do better?

As you pointed out in your review, our bound in Eq. (7) is a more interpretable version of the online lower bound introduced by Bui et al. (2017) using Panos et al. (2018) reparametrisation, so both methods share the same properties. The novelty in our method is that, unlike Bui et al. (2017), which used a fixed number of inducing points and heuristically retained 30% of the old ones, we propose a principled way to decide how many and which new inducing points to add to maintain the approximation quality.

Thank you for your thorough review and your suggestions. We appreciate your positive feedback.

Connection of our work with NNs

As we note in response to other reviewers, we view VIPS as a first step towards adaptive size in more general settings. Since GPs and NNs share structural similarities, the aim of Section 3.2 was to introduce these ideas as a foundation for future work on adaptive neural architectures. We are actively exploring such extensions in ongoing work.

It is unclear to me how this work could interact with alternative approaches, such as the expanding memory approach of Chang et al. (2023).

We believe our approach is complementary to Chang et al. (2023). Both approaches aim to retain a selected set of points (whether data or inducing points) to ensure a good approximation. Chang et al. (2023) achieve this by expanding their memory sequentially, adding a fixed number of data points at each step. In contrast, our method dynamically adjusts the number of inducing points to maintain a desired approximation quality. One promising direction could be to adapt our criterion to guide memory growth in their framework; this is an extension we are currently considering.

We will also take your other suggestions into account for the camera-ready version.

Thank you for your encouraging feedback and for recognising the relevance of our work. We appreciate your positive comments on the clarity of our writing and experimental design.

Q1: Optimizing the inducing points, rather than selecting them from the training data, may lead to better solutions. Is it straightforward to implement this in the current model formulation?

Yes, it is straightforward to implement with our formulation. As our method builds on the variational framework of Bui et al. (2017), which itself builds on Titsias (2009), the inducing point locations can be jointly optimised with the other hyperparameters of the model using the streaming variational lower bound, just as in the batch setting. In this work, we opt to select inducing points locations from the data for simplicity. That said, our framework is compatible with gradient-based optimisation of inducing locations if desired.

Q2: Related to the item Other Strengths and Weaknesses, is it straightforward to extend the current base prediction model to a (sparse) deep (multi-layered) Gaussian model?

The extension is not straightforward. In the deep case, there is no analogous bound to Titsias' collapsed bound, which the base model in Bui et al. (2017) builds upon. However, we are actively working on extending VIPS to deep models.

The proposed approach is not a generic model adaptation method but is limited to a specific model (sparse Gaussian Processes). Additionally, GPs and sparse GPs are limited in their ability to model highly nonlinear or non-Gaussian data distributions.

We appreciate your observation. While our current focus is on sparse GPs, we view VIPS as a first step towards adaptive size in more general settings. In our work, we provide a principled criterion that adjusts model size to maintain accuracy with incoming data. In particular, since GPs and NNs share structural similarities, we hope that the ideas introduced here can inspire similar mechanisms for adaptive neural architectures. We are actively exploring such extensions in ongoing work.