Continual Learning in Open-vocabulary Classification with Complementary Memory Systems

We appreciate the encouraging comments that the paper is quite strong in tackling a very challenging problem with very promising results and realistic experiments. We also are glad that you find the current version well-written and easy to read, though we will improve further based on your and other reviewers’ comments. Thank you for the opportunity to respond to your comments and concerns.

References to TreeProbe: We also could not find specific references to TreeProbe, though again we note its strong relationship to the well-established idea of Lazy Learning. It seems an idea worth discovering or resurfacing, as the case may be.

Dependence on CLIP: We agree that "zero-shot" loses some of its original meaning when using models that have been trained on Internet-scale data. However, the ability to handle arbitrary label sets using language-image embeddings is certainly useful, and the Radford et al. CLIP paper investigates (Sec. 5) and demonstrates (Fig. 17) that overlap with datasets accounts for less than 1% of zero-shot test accuracy. The relative performance between the original CLIP model(s) and our method is informative and controls for any data leakage that CLIP may exploit.

Code release: If accepted, we will release our code for our method and evaluation well before the conference.

Supervised Baselines: We’ll extend Table 2 in supplementary with per-task linear probe and fine-tuned baselines. We will also add the number of classes (total is 1034), and performance of a fine-tuned model on all data.

We appreciate the encouraging comments that the problem and tree-probe approach are interesting and regarding the diverse experimental validation. Thank you for the opportunity to address your concerns.

Experimental settings: We agree to clarify and add detail in the main paper (Sec 4.1, 4.2) and supplemental (S-3) about experimental settings. We also clarify below and in our response to Reviewer ShHL.

• Our experimental settings are designed to represent different practical requirements of continual learning. In each case, labeled training examples are received in batches.
• Data incremental (Fig 2a): the batches are uniformly randomly sampled, reflecting a case when data is received, e.g. from users, and labeled.
• Class incremental (Fig 2b): each batch contains all samples of a particular category, reflecting a case where the developer intentionally finds and labels examples of new classes to increase capability.
• For data- and class-incremental, experiments are done individually for each task, and accuracy is averaged across tasks. The candidate label set for a task is constant, so accuracy is measured over all classes, even for classes that do not yet have training data.
• Task incremental (Fig 2c): A single system is trained for each task sequentially, and test accuracy is averaged across all tasks at each stage.
• Flexible inference (Fig 2d): In the task incremental setting, we also measure performance when the candidate label set does not match provided examples (zero-shot, averaging over three held-out tasks), or is the union of all tasks’ candidate labels and labels from zero-shot tasks (union + zero-shot), or is a subset of all candidate labels and labels from zero-shot tasks (mix+zero-shot). These are highly challenging scenarios that require retaining the ability to predict over arbitrary label sets, as well as the label sets that correspond to training examples.

Paper organization: We will consider moving the problem definition before Sec 3, as suggested.

References: Thank you for these references, which we will incorporate. Although [1,2] have similar goals, they use attribute-based representations, while we build on language-image representations. As mentioned to Reviewer ShHL, reported results in those papers and the Radford et al. paper show that CLIP achieves much higher zero-shot accuracy than these methods reach even after training. We confirmed this in early testing as well. This is also posted to Reviewer ShHL: In our own test, under the generalized zero-shot learning setting which requires mapping learned representations to a union of seen and unseen labels, CLIP obtains the seen and unseen accuracy of 91.7 and 94.7 on the AwA2 dataset, respectively, largely surpassing the results (seen: 60.2, unseen: 77.1) obtained by [B] after training.

Discussion to prompt-based methods: We agree that prompt-based methods are an important direction to explore in future work. Currently, such methods are suited mainly for task-incremental due to the large computation required. Generally, our AIM approach to combine systems can apply to any specialized model, so larger and more tuned models such as the prompt-based models will predictably improve accuracy (at cost to slower training). Table 2 shows that using a large CLIP model, for instance, increases target task accuracy from 80.5 to 90.2.

We appreciate the encouraging comments that the direction is inspiring, problem is interesting and practical, and experiments are detailed. Thank you for the opportunity to address your concerns.

Experimental settings: We agree to clarify and add detail in the main paper (Sec 4.1, 4.2) and supplemental (S-3) about experimental settings. There are two branches of "open vocabulary" in the literature, one based on language embeddings and one based on attribute-category relationships. We clarify our meaning below and in response to Reviewer aTSX.

• Task: A task is defined by an image distribution and a set of candidate labels. The model’s input is an image and a set of candidate labels (without an explicit task identifier), and the output is one of the candidate labels.
• Target task: A task for which at least some training examples are received. We aim to improve performance on the original CLIP model for these tasks.
• Zero-shot task: A task for which no training examples are received. Prediction on zero-shot tasks is enabled using vision-language embeddings. We aim to retain performance from the original CLIP model on these tasks.

References: Thank you for these references. We'll incorporate them and others to improve our related work section. A major difference between many of the “continual zero-shot learning” or “generalized zero-shot learning” is that these methods, such as [B], rely on attribute-based representations, while we use image and language embeddings to predict novel/unseen labels. In reported results and our own tests, CLIP has much higher zero-shot accuracy (e.g. for the aPY and SUN datasets) than the continual attribute-based methods achieve even after training, supporting the idea of building on vision-language models for open vocabulary image classification. In our own test, under the generalized zero-shot learning setting which requires mapping learned representations to a union of seen and unseen labels, CLIP obtains the seen and unseen accuracy of 91.7 and 94.7 on the AwA2 dataset, respectively, largely surpassing the results (seen: 60.2, unseen: 77.1) obtained by [B] after training.

Comparison to ZSCL: We consider our performance with respect to the most closely related SotA ZSCL to be highly encouraging. Here’s why:

1. Shown in Table 3, our linear probe outperforms by 1.2% in Transfer, 1.7% in Avg, and 2.4% in Last performance. Our TreeProbe method outperforms by 1.2% in Transfer, 0.5% in Avg, and 1.9% in Last.
2. The Transfer performance reflects nearly perfect retention of zero-shot capability, which is also demonstrated in other experiments, while the forgetting experienced by ZSCL and WiSE-FT will tend to grow as more is learned due to their weight tuning strategies.
3. Simplicity/robustness: Our method outperforms ZSCL on their own test setup, despite that:
   • Their method fine-tunes; ours does not fine-tune.
   • They tune learning rate parameters for each task individually; we do no hyperparameter tuning.
   • Their method requires a large auxiliary dataset (ImageNet + ConceptualCaptions); we require no additional data.

Performance on some tasks: For particular tasks that especially benefit from fine-tuning, our system may underperform, but this can likely be mitigated in future work that explores developing stronger exemplar-based models while still limiting the computation required to incorporate new examples.