Towards guarantees for parameter isolation in continual learning

We are glad to hear you found our paper interesting. Hopefully, we can clarify your doubts with this answer, and improve your judgment of our work.

 

• "My major concern is about the definition of forgetting in equation (1), which is defined based on the training loss. "

  You raise a good point regarding the test loss. It is true that we establish guarantees for the training loss and, if the test loss significantly differs from the training loss, our guarantees are completely useless. However, as we know from statistical learning theory, the training loss converges to the test loss as the dataset size increases (keeping the model size fixed), therefore we can expect the train-test difference to be small for large enough datasets. And that is what we see in practice, for in all our experiments we measure forgetting on the test data (Section 5, Experimental Setup).

 

• "Another concern is about the implications of the theoretical results. Based on the theoretical results, the authors further propose an augmented version of SGD. But the effectiveness of this inspired design is only justified by comparing with vanilla SGD."

  Regarding the augmented version of SGD, SGD$^\dagger$, we have carefully stressed in the paper that we do not wish to contribute a new CL algorithm, and we only resort to the use of this modified SGD as a tool to test our theory. SGD$^\dagger$ has many problems. We cite from Section 5.3:

  “Overall, this training setup is suboptimal in many respects (performance, computation, memory) and it is designed specifically to validate the theoretical findings. Examples of efficient algorithms implementing the null-forgetting constraint are OGD, GPM and the other algorithms discussed, and competing with existing algorithms is not a goal of this study.”

  Nonetheless, we believe that our contribution has a value in that it progresses the theoretical understanding of continual learning algorithms, in a field especially dominated by empirical research. Our theoretical findings not only provide a unifying description of a set of otherwise distinct algorithms, showing that they are essentially equivalent (and abstraction is the goal of any theory), but it can guide principled development of new algorithms, perhaps more efficient and effective than the existing ones.

 

• (Question 1) "The authors claimed that minimizing the average forgetting is equivalent to minimizing the multi-task loss in section 1.1 "

  In the paper we say “The goal of continual learning algorithms is to minimise $\mathcal{E}(t)$ while learning a new task $T_t$, or, equivalently, to minimise the multi-task loss”. Importantly ‘while learning a new task $T_t$’, which means in formula minimising $\mathcal{E}(t) + L_t(\theta)$. Equation (1) is a straightforward definition of forgetting and we’re not the first to use it. It’s not clear in this setting what you mean by correlation or independence.

 

• (Question 2) "How do you define the task solutions in Assumption 2?"

  The “task solutions” are simply local minima of the task loss (as said in the sentences before the Assumptions).

 

• (Question 3) "Table 1 is difficult to understand. In what scale will the errors be considered as small?"

  We agree, it is confusing when the forgetting value is itself very small. In any case we would take $\mathcal{E}_o(t)$ as a scale reference there.

Thank you for your positive review and we appreciate the good quality of your questions. With the following, we hope to clarify all your earlier concerns.

• "The second-order analysis is restrictive to the case when the learning rate is small."

  A second-order Taylor expansion, like any local approximation of a function, naturally holds only within a given distance from its center. In our work we show that some parameter-isolation methods, like OGD, rely on a second-order approximation of the loss, and therefore are fundamentally limited by the locality of the approximation. This is an important result in our view, because it informs us of the inherent flaws of the algorithm. However, we can successfully apply our analysis to other algorithms such as GPM which elude the locality constraint by using different second order approximations along the optimization path (see Theorem 5 for details). Thus the 'small learning rate' constraint is not a weakness of our analysis, rather of some of the algorithms analysed.

 

• (Question 1) "You mention that parameter isolation strategies can provide stronger guarantees than regularization-based methods. Is the intention to stress that your analysis doesn't involve the optimization procedure?"

  Yes, that is correct. By design regularisation-based algorithms reach a compromise between the task loss and the regulariser. However, the extent to which they compromise in favor or at odds with the regularisation ultimately depends on the optimization process. On the contrary, parameter isolation methods enforce a hard constraint on the parameters, which holds independently of the optimization method employed. If you will, the difference between regularisation-based and parameter isolation is akin to the difference between regression with L2 regulariser and sparse linear regression.

 

• (Question 2) "Can you also give a brief comparison between your results and the guarantees of the regularization-based methods?"

  That’s an important point. Yin et al. provide theoretical results akin to ours in the space of regularisation methods. They carry out an optimization-based analysis, introducing smoothness assumptions and they provide convergence results for SGD with CL regularization and a generalization result (which is akin to a guarantee of optimality) holding in probability. In a nutshell, the regularization setup does not allow for as strong guarantees as the parameter-isolation setup (i.e. guarantees that hold regardless of smoothness, optimization algorithm or sample set size). The reason for that is that regularization methods enforce low forgetting as a soft constraint, while parameter isolation methods enforce a hard constraint on the parameters.

  Interestingly, our results align with those of Yin et al. for regularisation-based methods in showing that the constraint enforced by the algorithm is based on a second-order approximation of the loss. Thus regularisation and parameter isolation methods use the same information to prevent forgetting, however the former resorts to soft while the latter to hard constraints.

 

• (Question 3) "When we talk about the strict parameter isolations, does the network know the task domain index at training and inference time?" That is a good question. It ultimately depends on the specific method. For example, PackNet uses a task-specific parameter binary mask, and network expansion methods assign a different parameter set to each task. On the other hand, sparsity-based algorithms like SpaceNet do not need to know task identity information at test time, but use it while training.

(continued in the following comment)

We are quite surprised by your score and hope, despite the high confidence, that we can convince you to reconsider with this answer.

Weaknesses

• "At the end of Section 1.1, the paper claims that minimising average forgetting is equivalent to minimising the multi-task loss."

  Citing from the paper, we say “The goal of continual learning algorithms is to minimise $\mathcal{E}(t)$ while learning a new task $\mathcal{T}_t$ or equivalently to minimise the multi-task loss.” If you are referring to this sentence, perhaps you might have missed the ‘while learning a new task’ part, which is exactly what you claim to be missing?

 

• "In Section 5, the authors say, "we report forgetting in terms of accuracy", which I did not understand [...] What does this mean (because forgetting is not equal to accuracy)?"

  Yes, forgetting is not equal to accuracy. As defined in Equation (1), forgetting is the difference in loss after a change in parameters. However the loss is often not easy to interpret, so we instead quantify the results in terms of difference in accuracy after a change in parameters. We should clarify this better in the paper. Note that a precise definition has been given in the Appendix (Section B.0.1, Experimental Setup - Metrics).

 

• "I'm not sure if OGD and GPM are 'parameter isolation' methods, a claim that seems central to the title and framing of this paper."

  • You’re right in noticing that our definition of parameter isolation algorithms is wider than the convention in the literature. This is a point we should stress more in the paper. In a way, we are proposing a change in the vocabulary, and we do so based on the results of our analysis.
  • We define parameter isolation as algorithms partitioning the network parameters between tasks, and we show that OGD and GPM (and consequently all their variants) do it in a more sophisticated way than PackNet or Progressive Networks.
  • Essentially, we show that all these algorithms rely on the same mechanism (i.e. tasks parameters orthogonality) and therefore they can be seen as special cases of a unique strategy. Similarly, all regularization-based methods rely on soft constraints on the parameter change (determined by the regulariser) and replay-based methods rely on empirical, small-sample approximations of the previous tasks' losses.
  • Therefore labeling OGD and GPM (and their variants) as replay methods does not really reflect the nature of these algorithms, which is much more similar to that of PackNet than Experience Replay. However the connection to parameter isolation has not been ascertained before in the literature, and we accomplish this with Theorems 3-5 in this paper.
  • Moreover, several different categorizations of continual learning algorithms exist today, each of them using different terms. One may say that the continual learning terminology is for most parts still in progress, and should not be regarded as final yet.

 

• "Assumption 2 (combined with assumption 1) seems like a very strong assumption to me."

  It is. We do not claim that OGD is perfect, and we are well aware that it does not achieve zero forgetting in practice. Our theory can explain why that happens, which is precisely when Assumption 2 is violated. In fact, we even provide experimental data (Figure 3) to support this claim.

  In Section 5.1 we ask at what point this assumption breaks and we do not conclude that it always holds (which is naturally unrealistic). Additionally, we would like to highlight that later in the paper we do extend our theory to settings where we can drop Assumption 2, like GPM. Therefore our theory is not conceptually limited by the assumption.

 

• "In the first part (Table 1) the authors do not train until (close to) convergence, hence breaking assumption 1."

  In our first set of experiments our goal is to assess Assumption 2. Note that we do not need the gradients to be zero, i.e. Assumption 1, for the second order Taylor expansion (Equation 2). Therefore, it is not necessary in this case to train to convergence. Regarding the large standard deviations, that is the effect of using different seeds. As you might guess, the initialisation can play an important role in the geometry of the loss around it.

 

• "In the second part (Figure 2), I do not understand what reasonable values of the score or of r are."

  We have expanded on the interpretation of $s(r)$ in the answer to Reviewer W8iZ. To give you a short answer, a score $s(r)=1$ means that moving along an eigenvector, by a distance $r$, has the same effect as moving along a random other direction, by the same distance $r$, in the parameter space. Consequently, the values of $r$ for which the score is 1 identify regions of the parameter space where the quadratic approximation of the loss does not hold anymore.

(continued in the following comment)

• "My major concern is the significance or novelty of the submission."

  We are certainly not the first to analyze continual learning algorithms from a geometrical perspective and we do not claim to be pioneering the tools we use, such as the Taylor expansion. However, we believe our contribution to be unique because, as far as we know, we’re the first to have shown that a rather large number of methods (which we refer to as ‘parameter isolation’) follow the same general principle. A similar statement has been made previously for all regularization-based algorithms, but that’s a different set of methods.

  Also, we definitely agree that a favorable outcome of this line of research would be the development of principled algorithms. However, we believe that theoretical contributions such as ours are as valuable and necessary as empirical contributions, even more so in a field like continual learning that is largely dominated by empirical methods.

 

• "The authors analyze a few continual learning methods that already have sound intuition on their effectiveness, especially the OGD and GPM."

  Our theory applies to any method which, implicitly or explicitly, partitions the parameter space between tasks, e.g. through a linear projection (OGD), parameter freezing (PackNet), sparsity constraints, etc… So it could in principle also be applied to Wang et al., if the parameters used to learn the sequence of tasks were also partitioned. We do not cover all the algorithms in the parameter isolation family in our paper, because that would be unfeasible within the space limits, and it would be pointless given that new algorithms are proposed all the time. However, the examples we propose are meant to provide an intuition of how to apply the same strategy to other methods. In particular, for OGD and GPM (although they are ‘intuitively sound’) the connection to parameter isolation is not so clear, as shown by the extensive proofs backing the result. On the other hand, for ‘strict parameter isolation’ methods (Section 4.3) we do not go into the proof detail, because the link to parameter isolation is obvious. We would like to stress that in the respective papers, the authors of OGD and GPM do not establish any theoretical guarantees for the effectiveness of their methods, and that we are the first to obtain them.

 

• Regarding your question: $\mathcal{E}\_\tau(t)$ , is the forgetting of task $\tau$ after learning task $t$ (see Section 1.1, after Equation (2)). In the specific case of Theorem 2, we identify a constraint by which $\mathcal{E}\_\tau(t)=0$ for all tasks $\tau<t$.

We hope to have addressed all your concerns and to have better communicated the contribution of our work so as to improve your judgement of the paper. If you have any new questions please let us know, we’re happy to answer them.