Amphibian: A Meta-Learner for Rehearsal-Free Fast Online Continual Learning

We thank the reviewer for taking the time and review our paper. We provide responses to the reviewer’s questions/concerns below:

According to the reviewer - “the key idea of this work is to adjust the per-parameter learning rate” and “all the scaling is independently performed for each individual parameter”. These statements are not true. As we clearly mentioned in the paper that our method learns a diagonal scale matrix at each layer and scale (adjust) the gradient directions with this matrix accordingly. To elaborate on this point, let’s assume we have a fully connected layer of a network having weight, W of size $m\times n$. Here, $m$ and $n$ are the numbers of output and input neurons respectively. There is a total of $mn$ parameters in this weight. Instead of learning $mn$ learning rate parameters, for this layer, we learn a diagonal scale matrix of size $n\times n$, thus only learning $n$ parameters. (Table 7 in the appendix lists the actual number of scale parameters learned by our methods in different experiments.) Alternatively, we can think of this as gradients of a group of $m$ parameters are scaled by a common factor ($\lambda_i$), and there are $n$ such groups in $W$ matrix. As we adopt the concept of low-dimensional gradient space (please see section 3 in the paper for details) such scaling is possible. This low-dimensional gradient representation requires the introduction of basis vectors, $e_i$. Because, in conventional representation, the gradient space would be $mn$ dimensional for the weight matrix. However, in our low-dimensional representation, it is $n$ dimensional. The $e_i$ vectors describe the basis of this space. The scales, $\lambda_i$ are associated with each of these vectors, which are learned/computed from the alignment history along the basis directions (Equation 8, 9).

On the justification of method:

We are using a bi-level (inner-loop and outer-loop)/meta optimization method for CL. In inner loop, we take $k$ number of gradient steps, where each step uses a (non-overlapping) subset of data from the current batch $\mathcal{B_i}$. These inner loop updates intuitively simulate a ‘mini continual’ learning scenario. At the end point of inner-loop update the outer-loop loss on the whole batch $\mathcal{B_i}$ is computed. This loss indicates how well the current samples are learned and how much forgetting is there due to such (mini continual) learning. The outer-loop update then adjusts the weight to minimize forgetting and ensure good learning on the given batch. Now, to have learning synergy among different sequential batches, we use scale matrix that stores directional gradient alignment information. The direction along which samples in the online sequence have positive (negative) cumulative gradient alignment, model updates along those directions are encouraged (prevented). With extensive experiments with diverse datasets and architectures, we have shown the effectiveness of this method. Also, we show that loss on the previous tasks has minimal to no increase due to such updates which further justifies our method for CL.

Now, $\mathcal{|B_i|}=1$ ($k=1$) is not an ideal scenario for an meta-learning method like ours. This would mean there is no mini-CL sequence to observe in inner loop. Thus in the outer-loop no meaningful gradient information (for CL) will be generated for model update. For the same reason, the scale update would not be useful, as we won’t be able to measure and store any cross-sample gradient alignment information (Eq 8,9), which plays a key role in Amphibian. This also explains the alleged inconsistency between functionality of EWC and Amphibian. In $\mathcal{|B_i|}>1$ ($k>1$) case, however, Amphibian produced expected CL performance. In Section 6.2 (and in Figure 5(c)-(d)) we show how Amphibian and EWC become functionally equivalent methods for CL with an added constraint with hyperparameter $\beta$.

To enable Amphibian work with $\mathcal{|B_i|}=1$, a temporary memory buffer of size $B$ would be needed. Typically, $B$ would be of size of 5 to 10 samples. For each incoming example, the model would take an inner gradient step and that example will be stored in that buffer. Once the buffer is full, outer gradient will be computed on the entire buffer data and then the buffer will be cleared out immediately. This is how Amphibian would work in the reviewer-suggested truly/fully online setup.

On the confusing notations: We thank the reviewer for pointing this out. We will correct the typos (e.g. $i$ vs $m$) and restructure the notations (wherever applicable) in the revised manuscript.

On the technically incorrect statements: (1) We used the definition of online setting as per literature (La-MAML, A-GEM, ASER ) where training examples are provided as a series of batches. (2) Yes, Eq. (6) and 7 are equivalent under approximations (Appendix A). We will highlight this point in the manuscript.

We thank the reviewer for taking the time and review our paper. We provide responses to the reviewer’s questions/concerns below:

On the Weaknesses:

1. As per La-MAML we used the online (single epoch) learning setup where there is a glance hyperparameter. To have a fair comparison, in Table 1 we have reported results for all the other baselines in the same setup where multiple glances are allowed. Table 6 in the Appendix lists all the hyperparameters for these baselines, which contains the glance values as well.

2. In their implementation, La-MAML used ReLU operation in both inner and outer loops (Please see official La-MAML implementation [1] lines 55 and 99). In Table 1 we showed results for La-MAML (with rehearsal) and in Table 3 La-MAML (without rehearsal). Comparing these results we see that La-MAML with replay performs significantly better in terms of accuracy and forgetting mitigation. Since our method does not use any rehearsal and outperforms La-MAML (with rehearsal), learned diagonal matrices (unique component of our method) play a major role in preventing catastrophic forgetting.

3. The proposed method can also work in Class-IL online setup and the results are given already in the manuscript (in Figure 4, section 6.1).

[1] https://github.com/montrealrobotics/La-MAML/blob/main/model/lamaml.py

We thank the reviewer for taking the time and review our paper. We provide responses to the reviewer’s questions/concerns below:

Answer to Q1: The main differences between our method (Amphibian) and La-MAML are: (1) La-MAML uses a replay buffer for storing old data, whereas Amphibian does not use any replay buffer. Thus La-MAML is a rehearsal-based method whereas Amphibian is a rehearsal-free method. (2) LA-MAML learns per parameter learning rate using data from both current and past tasks, whereas Amphibian learns layer-wise diagonal scale matrix using only the current batch data. Thus, the number of extra learnable parameters (memory overhead) in La-MAML is very high compared to Amphibian (Please see Figure 4(c)). (c) Amphibian minimizes a meta-objective that encourages alignments of gradients among given data samples along selected basis directions in the gradient space (Eq 7), La-MAML optimizes a different meta-objective.

Answer to Q2: We adopted a low-dimensional gradient space representation in each layer of the network (please see section 3 for details). For each basis describing this space, we wanted to assign a scale by which incoming gradients will be modified during CL training. For this purpose, we introduced a diagonal scale matrix (in Eq. 4,5).