From Categories to Classifier: Name-Only Continual Learning by Exploring the Web

We address the ethics concern of copyright in training models and data release raised by the reviewer:

(a) Copyright: We kindly request the reviewer to differentiate between motivation for future commercial applications based on recent trends in selected countries detailed in the introduction, with the research work itself. To the best of our knowledge (we are not legal experts), training classification models on copyrighted data for this research paper is perfectly legal practice and covered by copyright laws in UK [1] and EU [2] for academic research purposes, like most other papers which train models on vision datasets.

(b) Data Release: We clarify that we will only distribute links of the images for all C2C datasets with the class-names, including EvoTrends for reproducibility. Distributing links of images is a perfectly legal practice and does not violate copyright laws.

We hope this alleviates reviewer concerns. We added a Section E in Appendix highlighting these points.

[1] https://www.gov.uk/guidance/exceptions-to-copyright#non-commercial-research-and-private-study

[2] https://digital-strategy.ec.europa.eu/en/library/copyright-digital-single-market-brochure

Thank you for your insightful review. We appreciate your recognition of our paper's clarity, thorough experimental analysis, and discussion on web-supervised limitations. Your acknowledgment of the significant effort in our engineering and implementation is highly encouraging. We will address the concerns raised in the following responses:

W1 and Q1 [Research Contribution & Novelty]. We respectfully disagree with the reviewer and would like to push back strongly on this claim. We claim that:

• [Webly supervised learning for learning targeted tasks.] We adapt webly-supervised data to be used in downstream tasks. To the best of our knowledge, webly supervised learning has been primarily used as a pretraining task in past literature. Use in downstream tasks often entails targeted collection of data instead of a broad sweep across the internet. See recent work "Internet Explorer: Targeted Representation Learning on the Open Web" for a detailed outlook.
• [Impactful Solution.] Previous works often introduce algorithmic novelty by designing ad-hoc data collection components using large vision-language models; we show that revisiting webly supervised learning is far better than current hot trends of using retrieval based approaches and synthetic data generation in a name-only classification setting.
• [Extensive Engineering.] We alleviate past shortcomings in webly supervised learning by introducing new safeguards, optimize the scraping pipeline, re-examine various components for webly-supervised learning to name-only settings such as data sampling, effectiveness of various search engines, and that curation of webly-supervised data is unnecessary as linear probes are insensitive to this noise.
• [Finding shortcomings in continual learning and alleviating them]. We uncover important shortcomings in continual literature along with extensively engineering a simple method that can have a major real-world impact.
• [Analysis of why our method works, and what is still missing]: We provide not only strong results, but also isolate the reasons for improvement from our method by presenting an analysis on the impact of scale, effect of noisy samples, effect of model architectures, etc in our work with counterintuitive findings. We provide a detailed qualitative and quantitative overview of distribution shifts in the continual datasets in Appendix D to highlight the key bottlenecks in continual name-only learning.

[Humans involved in webly-supervised learning] We do not understand the question. We acknowledge that internet is the creation of human labour and algorithmic systems. We believe it is quite likely that human experts used Flickr to initially filter samples and not vice-versa. We are not aware of any mechanisms to index these datasets back explicitly. Computing a match fraction is flawed in the sense it does not isolate the samples indexed back, rather than initially filtered which is perfectly valid.

[Expert-in-the-loop]. Thanks for bringing this to our notice. We have cited it along with other active learning approaches as they are relevant works! However we note the following:

• SuS-X and Neural priming have a similar nearest neighbor based pipeline (but no human labels in loop) with comparable or higher gains to a zero-shot CLIP model to our best knowledge. Exact comparisons between these methods was unfortunately not possible as the authors of aforementioned expert-in-the-loop work could not share their collected data.
• We, in turn, outperform both SuS-X and Neural priming consistently by very wide margins across many downstream datasets and across architectures.

Thank you for your valuable feedback. We are grateful for your recognition of our paper's structure, and real-world impact of reducing the time and expense associated with manual annotation and for acknowledging our novel paradigm of name-only continual learning. We address the concerns raised below:

W1. [Not Innovative]: We respectfully disagree with the reviewer and would like to push back strongly on this claim. As we detailed in our Introduction section, and highlighted by Reviewer XoLU, 7ydL we highlight that:

1. Study a Pressing Problem: Continual learning methods traditionally ignore training data creation which creates a huge cost bottleneck. We bring this to notice of the community, formalize this problem and propose a way to alleviate this bottleneck. Our solution is applicable across an extensive range of continual learning scenarios such as class, domain and time-incremental.
2. Impactful Solution: Even in the static name-only classification scenario where we can compare fairly with alternative solutions proposed in literature, we move the field forward by very large margins of performance (10-25%) where other approaches listed in Table 3 and Appendix C published in CVPR’23, ICCV’23 and NeurIPS’23 improve upon previous works by 1-3% margins.
3. Analysis of why our method works, and what is still missing: We provide not only strong results, but also isolate the reasons for improvement from our method by presenting an analysis on the impact of scale, effect of noisy samples, effect of model architectures, etc in our work with counterintuitive findings. We provide a detailed qualitative and quantitative overview of distribution shifts in the continual datasets in Appendix D to highlight the key bottlenecks in continual name-only learning.

We request the reviewer to reconsider and not dismiss our hard work with a claim of “not innovative enough”.

W2. [Impact of noisy samples not studied]: We acknowledge the reviewer's perspective and clarify that our work already provides an extensive analysis on factors like: Scale in Table 2, Noisy samples across different dimensions like duplicity and irrelevant samples in Appendix B.1 (as requested by the reviewer), different architectures in Appendix B.1, different search engines in Appendix B.1, class-balancing as web samples are highly imbalanced in Appendix B.1. We hope this analysis alleviates the concern of the reviewer.

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

Thank you for your valuable feedback. We are grateful for your recognition of our paper's quality and for acknowledging our as the first work to introduce the continual name-only setting. Your appreciation of the in-depth analysis and comparisons in our work is highly encouraging. Additionally, we are happy that you liked the merit of EvoTrends as a more realistic approach to class-incremental continual learning.

We address your concerns below:

W1 [On Continual Learning Algorithm]

Q1. [Retraining] We provide the following details to give more clarity: At each timestep t, the classifier (a single linear layer) is simply initialized based on the linear model at task t-1, and updated quickly with experience replay on past samples with random sampling. There are no extra bells and whistles.

Why do we use this? This was shown to be state-of-the-art in scenarios with limited compute but unrestricted memory (Prabhu et al, 2023a). We follow the same setup, fixing the number of FLOPs and hence computational cost into tight, normal and relaxed regimes.

We agree this could be expensive, hence additionally present the NCM and approximate kNN baselines which take the classifier at task t-1 and simply add the new samples to it. Hence, they can operate on “tight” computational budgets being some of the cheapest approaches to implement.

However, due to the linear classifier being updated only on extracted features and no prior samples are stored this process is relatively fast in practice. Compared to finetuning the whole backbone, training a linear classifier is extremely efficient, emphasized below.

Q2. [Evaluation of Domain/Time-Incremental Setting] We were not sure if we understood the question correctly.

• How do we query the web? As detailed in Section 5.1, for domain-incremental setting, we query the web with class-names + domain-names, but consider them in the same class according to class-names. For time-incremental settings, we query the web using timestamp queries to limit time ranges and only consider class-names for the labels.
• How is catastrophic forgetting alleviated? RanPAC [1] and ACM (Prabhu et al., 2023b) already show the effectiveness of NCM and ACM respectively for specific domain and time incremental scenarios. Inspired by this, we benchmark those methods in a name-only continual learning setting and find that kNN outperforms linear classification which outperforms NCM in a name-only setting. The effectiveness of simple approaches is quite underrated in tackling catastrophic forgetting is quite underrated in the continual learning domain.

[1] RanPAC: Random Projections and Pre-trained Models for Continual Learning

Q3. [Finetuning] This is a great question! While finetuning achieves better performance, we clarify that it is dramatically more expensive. We could not satisfactorily train it on our continual learning setup, even with 10x computational resources compared to the relaxed compute budget. Hence, its usefulness is very limited. Hope this addresses the concern of the reviewer.

We hope we were able to clarify all your concerns in W1.

[W2] Analysis results would not translate to a continual learning setting.

• We are not sure why the ablations on noise, multiple web engines, data cleaning and class balancing etc would not translate as they are quite dataset independent. We did a detailed analysis of the distribution shift to detail the precise domain gaps between web datasets and annotated training data in Appendix D.

What did we aim for in a continual learning setup?

• We had a high bar for compute efficiency so we focused on those aspects. Note that latest approaches which use stable diffusion models to generate images or retrieve relevant images from a 5B scale dataset by text are far more expensive than the budgets allowed in our relaxed setting.

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

We thank the reviewer for appreciating our exploring of webly-supervised data for fine-grained classification and continual learning, the quality of our work, and our provided insights on the power of webly-supervised data for visual learning tasks.

We address your concerns below:

W2. [On Restricted Memory]. This is a great question. We clarify here that while most prior continual learning approaches have the assumption that data cannot be fully stored either due to privacy concerns, when this happens is underspecified (a memory buffer is used to store a limited number of samples of choice). Our framework allows us to study the validity of this data constraint – Since we retain only links of images, we can access past data whenever we can by redownload the links. If the content is removed from the link, we can no longer access it. We could redownload 100% of the data within the span of a few weeks when we ran the final experiments. Longer timescale works of (Cai et.al., 2021) find that they could re-download 95+% of the links last collected in 2014 entailing that links typically do not get removed.

Hence, we rely on the setup constraints introduced in prior work which argues against restricted memory (Prabhu et. al, 2023a) claiming a wide range of real-world constraints can have relaxed memory restrictions. We clarified this in our paper draft in Section 5.1 Training models.

W1 [Added comparisons with CL Approaches]. We clarify here that previous work (Prabhu et.al., 2023a) showed that in computationally budgeted setup with relaxed memory constraints, traditional continual learning methods fail to be effective and simple experience replay baseline performs the best. Hence, we did not study the past methods, but additionally included baselines such as kNN and NCM which have been shown to work well in similar settings (Prabhu et. al., 2023b). However, we highlight that we do not propose any new continual learning methods but focus on ways of collecting training data, extensive comparisons with CL methods is somewhat orthogonal to the focus of our work.

W3 [Comparison with zero-shot CLIP approaches unfair]. We agree with the reviewer. We additionally compare with methods which also collect data from large datasets like LAION5B leveraging CLIP models in Table 3, alongside the data-free methods. We compared data-free methods as it is a valid method of performing name-only classification, but as reviewer notes and we present in Table 3, methods which collect training data outperform methods which do not use training data. We note that we outperform the latest methods which also collect data along with large models (presented in CVPR ‘23, ICCV’23 and NeurIPS ‘23) by large margins.

W4 [Continual Learning Results worse than MA, Quality Reasons]. This is a great question. We clarify here that the performance on manually annotated datasets serves as an upper bound and not a fair comparison as they require a very expensive curation process, the collection has biases making them well-aligned with the test distribution as both sets are created after labeling by subsampling data. Reproductions of the test set creation of CIFAR10 and ImageNet [1] show that just by recreating the test set following the same procedure as originally followed, they observe 5-10% drops in performance of models despite no training data drifts etc. We updated the draft to highlight this point.

In Appendix D, we investigate the performance drop between our webly-supervised data and annotated data in detail and find distribution gaps between the manually annotated test set and our training set. For example, for CLEAR time-incremental dataset we show that the buses there are interior of buses whereas when we scrape samples we scrape exterior of buses. Similarly, for PACS, manually annotated sketches are simple quick doodles compared to webly-supervised sketches which are pencil sketches. This indeed creates an extra gap between our training data and the test data.

[1] Recht et. al, Do ImageNet Classifiers Generalize to ImageNet?

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

We thank you for the constructive feedback and the positive aspects you highlighted in our work. We are pleased that you found our approach in reducing manual annotation costs, the outstanding results in webly-supervised learning under both single-task and continual learning, and the application of our methods to real-world scenarios useful. Your recognition of our counterintuitive findings on learning with noisy data and the thoroughness of our evaluations, especially in the context of multiple fine-grained vision datasets and cost-aware continual learning, is motivating.

We address your concerns below:

W1. [Access to Past Data]. This is a great question. We clarify here that while most prior continual learning approaches have the assumption that data cannot be fully stored either due to privacy concerns, when this happens is underspecified (a memory buffer is used to store a limited number of samples of choice). Our framework allows us to study the validity of this data constraint – Since we retain only links of images, we can access past data whenever we can by redownload the links. If the content is removed from the link, we can no longer access it. We could redownload nearly 100% of the data within the span of a few weeks when we ran the final experiments. Longer timescale works of (Cai et.al., 2021) find that they could re-download 95%+ of the links last collected in 2014, entailing that linked images typically do not get removed.

Hence, we rely on the setup constraints introduced in prior work which argues against restricted memory (Prabhu et. al, 2023a) claiming a wide range of real-world constraints can have relaxed memory restrictions. We clarified this to our paper draft in Section 5.1 Training models.

W2. [Cost Effective Safeguards]. We apologize for not detailing this in more detail, but emphasize that we had seriously considered this ethical concern due to past mishaps (Birhane and Prabhu, 2021) and tried our best to alleviate it. In our paper we use two particular safeguards: (1) Using safe-search filtering (as mentioned) and (2) Making sure that the category-name is written to ensure no ambiguity in the search query that could be understood as a non-safe content. For example, we would correct "african ass" to "african wild donkey" or "Equus africanus" to avoid inadvertently collecting explicit content. We note that adding the auxiliary suffix as stated in "How to design queries the web?" also greatly helped filter unwanted contexts cheaply. We manually inspected a few hundred of the downloaded images for data quality with random sampling and found that strict safe-search filtering in web search engines by itself is extremely effective. We updated the draft with the second part in Section 3 Step 1 and added an explicit Ethics subsection in Appendix E.

W3. [ResNet vs ViT]. In Table 8 (Appendix B), we show the impact of various network architectures and pretraining algorithms. ViT based models did always outperform the ResNet models, including on the Cars dataset– we are not sure if we understood the the reviewer's concern correctly.

W4. [Typo]. Thanks for bringing it to our notice, we fixed it in the updated draft.

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.