Temporal-Difference Variational Continual Learning

Thank you for the review! We appreciate that you found our work well-written and clear, the empirical results supportive, and our introduced benchmarks a potentially valuable contribution. You raised great questions, which we address below:

Q1 Are Eqs 2 and 3 equivalent? Is Eq 3 a lower bound of Eq 2? What is the source of approximation errors?

A1 We argue that Eq 3 is equivalent to maximizing a lower bound of the marginal likelihood (evidence) and not a lower bound of Eq 2. And the optimization w.r.t $\theta$ for both objectives is equivalent. We show this with an ELBO derivation:

\underbrace{\mathscr{D\}_{KL}(q(\theta) \mid \mid \frac{1}{Z\_{t}} q\_{t-1}(\theta)p(\mathcal{D}\_{t} \mid \theta))}\_{L\_{2}(\theta)} = \mathbb{E}\_{q(\theta)}[log \frac{q(\theta)}{q\_{t-1}(\theta)} - log(p(D\_{t} \mid \theta)) + logZ\_{t}] =\underbrace{\mathscr{D}\_{KL}(q(\theta) \mid \mid q\_{t-1}(\theta)) - \mathbb{E}\_{q(\theta)}[log(p(D\_{t} \mid \theta))]}\_{-L\_{3}(\theta)} + logZ\_{t}

where $L_{2}$ and $L_{3}$ are the terms in Eqs. 2 and 3. Note that minimizing $L_{2}$ is equivalent to maximize $L_{3}$ as $Z_{t}$ is constant w.r.t the optimization parameters. Since $L_{2} \geq 0$ (it is a KL divergence), then $L_{3} \leq log Z_{t}$.

Eq 3 is effectively a lower bound on the evidence. Indeed, the approximation error is due to the lower-bound, but the gap is exactly quantified by the KL term in Eq 2. We identify two error sources: computing the objective itself (due to sampling errors or biases from previous estimates) and the inherent choice of the variational family (which introduce biases for the subsequent steps). We note that deriving the ELBO with Jensen's inequality is equivalent, but the presented derivation directly quantifies the lower bound gap.

Q2 Implicit (VCL) vs Explicit (TD-VCL) Regularization in previous posteriors and the effect in error compounding (Points 2 and 4)

A2 Indeed, both VCL and TD-VCL optimize the same KL div in Eq 2. The difference lies in how the approximation errors across successive steps interact: we refer to our geometric interpretation in Figure 1. If the errors do not follow a particular pattern in the parameter space, we argue that explicitly regularizing in previous posteriors exerts a corrective influence ("canceling out" errors). Naturally, if all posterior estimates exhibit similar error patterns (intermediate errors in the same "direction"), then TD-VCL and VCL would behave and compound errors equivalently. We argue this needs one to adversarially pick a particular variational family distribution and optimization algorithm. In practice, however, we observe that there is indeed a corrective influence, and TD-VCL objectives work better, as shown by the presented experimental validation.

Q3 Setup where tasks change quickly

A3 We assume "tasks changing quickly" indicates few data points per task. This "low-data" regime should be handled by the Bayesian framework itself – the sequence of posteriors would be closer to the initial prior, regardless of the variational objective adopted. The prior encodes all knowledge we know a priori from the tasks. It is better to be closer to the prior, as overfitting to a particular task (like what MLE would do) can be very detrimental for other tasks, harming plasticity. Thus, the posteriors might not lead to good downstream performance due to the lack of data, but the posterior is still useful by not allowing plasticity loss. The Bayesian model also expresses epistemic uncertainty, which gives another layer of interpretability to the predictions that may be leveraged to prevent a very uncertain model from making bad decisions in these scenarios.

Q4 Intuitively, tasks presented earlier should lead to stronger catastrophic forgetting. Then why does TD-VCL prioritize more recent posteriors?

A4 Your intuition is often correct and supported by the findings. Yet, constraining the current posterior to a past posterior $q_{t}$ goes beyond constraining to the knowledge about task $t$ but comprises the whole history of precedent tasks $t, t-1, ...$. The recursive property in Eq 1 allows information from all past tasks to flow up to the posterior estimation. Constraining harder to an older posterior might help the corresponding task and previous ones, but also disregards subsequent ones. Older posteriors are not "aware" of newer tasks, while recent ones are conditioned in a longer history. Thus, it makes sense to give more weight to recent estimations.

TD-VCL actually constrains in many posteriors that are conditioned in an older task. At timestep $t+1$, Task $t$ is only accounted in posterior $q_{t}$ while Task 0 is accounted in all posteriors. This leads to a stronger constraining on an older task, which does not require a stronger constraining to the posterior of the corresponding timestep. This is a nice property that VCL does not enforce explicitly.

Thank you for your review! We appreciate the recognition of our theoretical derivations, extensive hyperparameter settings, and ablation studies. We're also grateful that you found our work innovative in combining TD learning ideas with Variational Continual Learning and demonstrating a strong grasp of Bayesian CL literature. We aim to address your concerns below:

Q1 The work should expand the experimental evaluation to include more challenging benchmarks (e.g., CIFAR100, TinyImageNet); current experimental evidence is limited to PermutedMNIST/SplitMNIST and reports lower accuracy than prior work.

A1 We highlight that our current paper does present an experimental evaluation on CIFAR100 and TinyImageNet. We refer to Tables 2/3 and Appendix L for the results. TD-VCL attained superior performance against other methods, as discussed in Section 5.1. As the reviewer stated, these are challenging benchmarks and should provide good experimental evidence to support our claims.

We also clarify that our experimental evidence goes beyond Permuted/SplitMNIST. In fact, we improve upon these benchmarks, introducing novel, more challenging versions that impose memory and architectural restrictions (namely Permuted/Split/SplitNotMNIST-Hard). As highlighted by reviewer c7SD, this is a potentially valuable contribution to the field. Our work actually reports higher accuracy than prior methods in all considered benchmarks, as shown in Tables 1 and 3.

Q2 The derivations assume that the normalization constant is independent of the variational parameters, which could lead to biases.

A2 By definition, the normalization constant (evidence or marginal likelihood) is independent of the parameter distribution, as these are marginalized out: $Z = p(D) = \int_{\theta}p(D_{t}\mid \theta)p(\theta)d\theta$. This is a crucial aspect for Variational Inference (VI) and for deriving an evidence lower bound for tractable objectives, including variational CL. As also explicitly stated by the VCL paper (Sec. 2.1) [1], "Zt is the intractable normalizing constant of $p^{*}_{t}$ and is not required to compute the optimum". Assuming a proper choice of variational distribution and optimization procedure, the learned posterior may theoretically achieve zero KL divergence w.r.t. to the true posterior, both in VCL and TD-VCL variants.

One important clarification is that, at timestep t, the optimization is w.r.t the variational distribution $q_{t}$, which does not influence $q_{t-1}$. This "prior" $q_{t-1}$ (and any previous posterior estimate used) is a fixed distribution for this optimization. There is no backpropagation through time.

Q3 Quantifying the Gaussian mean-field (MF) approximation error.

A3 The Gaussian MF approximation is standard in VCL objectives [1-3] and widely used in VI for its tractability and convenience. Assuming no optimization error, the approximation error from the choice of distribution family is quantified by the KL divergence between the learned variational distribution and the true posterior. Statistically quantifying this divergence in VI remains an active research area [4], often requiring simplifying assumptions about the true posterior. While valuable, this is beyond our scope — we focus on the algorithmic work of deriving a new tractable optimization objective for variational CL and empirically demonstrating its improved posterior approximation in downstream predictive tasks. Finally, prior work shows that for deep networks (our case), MF approximation is not too restrictive, and the bigger your model, the easier it is to be approximately Bayesian [5].

Q4 L’Hôpital’s rule and necessary regularity conditions.

A4 Our only use of L'Hopital's rule is in Appendix E. The "key" derivations of our proposed objective are in Appendices A/B/D, which do not use it. In Appendix E, we apply the rule to two functions. They are in the form $f(x)/g(x)$, where the numerator and denominator are differentiable and $g'(x) \neq 0$ in the considered interval (0, 1). The limits in the original form lead to the indeterminate form 0/0. Lastly, the limit of $f'(x)/g'(x)$ exists. Thus, we may apply the rule in both considered functions.

Q5 Is it possible to extend the convergence properties or derive error bounds from TD learning to TD-VCL?

A5 As in Sec. 6, we do not claim that the TD-VCL objective configures an RL algorithm or is equivalent to the Bellman Equation. Yet, we believe that the connections presented in the work inspire extensions in this direction. We hypothesize that it is possible to formally define a TD-VCL Operator, analogous to the Bellman Operator, and investigate contraction properties that motivates further work in statistical guarantees for posterior evaluation/improvement. Nonetheless, as stated in the paper, we left it as future work.

References are in Reviewer zpmQ's response.

Thank you for your review! We appreciate that you recognized our contributions (in formalism and empirical validation), found our ablations convincing, and proposed benchmarks detailed and justified. We aim to comment/clarify some of the raised points below:

Q1 Adoption of previous protocols to compare with other CL methods

A1 We understand the value of previous protocols to establish a more direct comparison. Nonetheless, we found the previous setups commonly used in Bayesian CL works not very challenging since they do not impose the memory and architecture restrictions, which would not lead to a proper setup for evaluating Catastrophic Forgetting. Still, the remaining configurations are equivalent, including for the harder benchmarks like CIFAR100-10/TinyImageNet-10. Furthermore, we make sure to adopt strong Bayesian CL baselines [1-3], controlling several aspects of the training and tuning procedures to be fair among the methods, as detailed in Appendix F. As our goal is to advance continual learning in the Bayesian framework, we believe our followed protocol is reasonable and supportive of our claims.

Q2 Comparing with other families of Continual Learning Methods

A2 We agree that providing a more exhaustive set of CL baselines would provide a better perspective on the current CL research landscape. However, CL research spans several directions, adopting different assumptions and desiderata. We opted not to broaden the scope too much. Otherwise, it would be really hard to control the experiments and perform fair comparisons. Alternatively, our work keeps baselines consistent in these terms, which allows us to make direct claims about the impact of the proposed objective. Also, most methods explore orthogonal design choices (e.g., architecture, memory, regularization). Given the flexibility of our objective, it can be directly combined with them, as illustrated in Table 3.

Lastly, as the reviewer stated, our work has a particular goal of advancing Bayesian methods for CL. We highlight that the Bayesian framework follows a principled approach that allows the development of uncertainty-aware models, which is crucial for robust, safe Machine Learning. These are capabilities that most other methods do not provide, even if they present better predictive performance in some scenarios.

Q3 Presenting other metrics for CL.

A3 We agree this would provide a more complex analysis of the algorithms. We opted to follow the standard metrics in the bayesian CL literature [1-3], which allows us to evaluate downstream performance and Catastrophic Forgetting directly in order to support our main claims. Nonetheless, incorporating additional, more granular metrics is an interesting direction for future work, and we appreciate the reviewer's recommendation.

Q4 Why is the connection between TD methods and VCL useful?

A4 We argue that this connection allows us to observe the variational CL problem setting with the lens of bootstrapping/credit assignment. This opens several venues to leverage TD methods developed by RL research, configuring the posterior search as a structured problem of value estimation. Furthermore, it allows us to potentially extend the theoretical analysis of variational CL methods with tools from RL theory. We believe our work is just a starting point that identifies an interesting intersection of both areas with encouraging experimental results.

Q5 How reliant is the method in knowing the task boundaries?

A5 The problem setting adopted in this work (and in the considered baselines) assume the tasks provided with clear boundaries. Still, in a broader case with unknown boundaries, we may have different tasks mixed on the same timestep. For the optimization objective, we believe the method should still perform well as long as likelihood estimation is feasible. Ultimately, it would be a multi-task learning situation. The only potential concern we anticipate is the potential negative transfer effect among tasks in the same timestep.

Q6 Which of the $n$ posteriors are used when sampling an example from dataset $t-k$ from the replay buffer?

A6 At a timestep $t$, all predictions are performed with the current posterior $q_{t}$. Past posteriors are frozen and only used for regularization.

Q7 Other comments/sugestions

A7 Thank you for highlighting them, we incorporated the fixes in our draft.

Rebuttal References

[1] Nguyen et. al. Variational Continual Learning. ICLR, 2018.

[2] Ahn et. al. Uncertainty-based Continual Learning with Adaptive Regularization. NeurIPS, 2019.

[3] Ebrahimi et. al., Uncertainty-guided Continual Learning with Bayesian Neural Networks. ICLR, 2020.

[4] Katsevich et. al. On the approximation accuracy of Gaussian variational inference. The Annals of Statistics, 2023.

[5] Farquhar et al. Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations. NeurIPS, 2020.