TiC-CLIP: Continual Training of CLIP Models

We thank the reviewer for their positive feedback on our work. We are glad to see the reviewer appreciating the importance of our continual learning benchmark and our experimental protocol.

No method, only benchmarking. The paper is commendable in its focus as a benchmark study … However,... incorporating … related works such as Continual-CLIP [1] and … "Robust fine-tuning of zero-shot models" [2]. Including these comparisons … highlighting … effectiveness in contrast state-of-the-art approaches.

We thank the reviewer for their appreciation of our benchmark study. We would like to clarify that we indeed perform experiments with methods close to both the references the reviewer pointed out. In particular, we perform experiments with Patching (Ilharco et al., 2022) and LwF (Li & Hoiem, 2017; Ding et al., 2022). To clarify: (i) We employ Patching which is a follow up of Wise-FT that adapts Wise-FT to continual learning settings where we observe more than 1 task in sequence; (ii) We adapt LwF for multimodal models which closely resembles the method in Zheng et al. 2023 (equation 4 performs LwF on both text and vision backbone).

No fine-tuning, only pre-training. An additional area of improvement lies in the paper's scope, particularly concerning the fine-tuning stage … Specifically, investigating how … checkpoint at a given time transfers to standard vision benchmarks, beyond simple retrieval tasks … This consideration is essential for gauging the real-world applicability and adaptability … .

We perform zero-shot evaluations with 28 existing standard vision benchmarks to evaluate our CLIP models (results are in Table 2 and the standard 28 datasets are listed in Table 4). Our observations highlight that continual pretraining improves on standard tasks (see Figure 1-right and Table 2). Moreover, existing work highlights that zero-shot evaluation is strongly correlated with linear-probe performance (Gadre et al. 2023; Figures 16,17 in Section O). This finding, together with our zero shot evaluation results on 28 datasets, highlight the real-world applicability of continual pretraining for linear probing evaluations.

If we misunderstood your comment, we request the reviewer to provide clarification. We would be happy to run additional experiments if the reviewer has suggestions. We are also happy to run linear probing experiments if the reviewer suggests that these comparisons would strengthen the paper.

We would also be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

We thank the reviewer for their feedback. We are glad to see that the reviewer finds our continual learning benchmarks as non-trivial contribution.

This benchmark lacks various types of continual learning methods [1]: elastic weights consolidation methods, progressive neural network methods, dynamic architecture methods, etc. Therefore, the experiments…is relative weak and insufficient.

While we included references to these papers, we do not compare with other methods such as elastic weights consolidation methods and progressive neural network methods because these methods significantly increase the computation cost (refer to Appendix A.1). Several existing works argue that in constrained-compute scenarios, these methods perform poorly than other CL methods, e.g., LwF and Patching, that are computationally more efficient (Ilharco et al. 2022, Wallingford et al. 2023). We did compare with LwF which often performs empirically better than or similar to EWC (Zheng et al. 2023, Ni et al. 2023). Our findings highlight that this method increases the compute and doesn't improve over our baselines.

Moreover, in our work, we show that simply by increasing the compute budget by 1.2$\times$ for cumulative baseline, we can bridge the gap completely and achieve performance close to or better than the Oracle method. In contrast, methods like EWC significantly increase the compute budget, i.e., at least by a factor of 2 (since we need to compute gradients for two models) excluding the time to estimate the eigenvalues which can also incur significant costs (Ding et al. 2022). We also thank the reviewer for sharing the reference on the toolbox. We have now included this in our work.

This paper lacks … vision-language models solving continual learning. Vision-language models enable various novel model tuning paradigms, such as prompt tuning, parameter-efficient tuning, … aspects are ignored … then this benchmark is a bit over-claimed.

We would like to clarify and further highlight the distinct nature of our work in comparison to existing research. Unlike the majority of prior studies which primarily focus on task/domain/class incremental setups, our work diverges significantly. In these previous works, such as those by Ni et al. 2023, Ding et al. 2022, and Zheng et al. 2023, the emphasis has been on handling small-scale downstream task data, typically of a very small order than pretraining dataset size. These methods tend to produce models specialized in individual domains. However, in our work, we focus on pre-training (general knowledge acquisition) of the base model with continually evolving internet data. We create the first benchmark with 12.7 billion timestamped img-text pairs that enables systematic study of continual pretraining of CLIP models. This benchmark paves the way for a systematic exploration of the continual pretraining methods in CLIP models.

While fine-tuning a limited set of parameters may be an effective strategy for tailoring a model to a confined dataset from a specific domain, this approach is less viable for maintaining and updating "general-purpose" foundational models (e.g., CLIP) with dynamically evolving internet data. In our setup, the proportional increase in new data relative to the pre-existing dataset necessitates comprehensive model updates.

(New result) As per your suggestion, we performed an additional experiment to assess the efficacy of updating selected model parameters versus full-scale fine-tuning with sequential training on the TiC-YFCC dataset. In particular, we compare LiT [1] where we perform full-finetuning on the first time step and then perform LiT for subsequent time steps on Tic-Datacomp (M). We observed that while updating a few parameters does avoid catastrophic forgetting on old timestamps, this happens at the cost of reduced performance on all tasks. For instance, the final Imagenet performance drops substantially from 24.0 (w full finetuning) to 7.3 (w LiT). This highlights limitations of finetuning only a small number of parameters while trying to keep general purpose foundation models up to date.

Moreover, we would like to emphasize that the main goal of the paper is NOT to introduce a new continual learning technique, rather introduce the first benchmark for continual pretraining where we pretrain foundation models as internet data evolves. For this setup, we present extensive results using standard continual learning approaches that are suitable for pre-training and highlight a simple method that enables continual pretrain of CLIP models without retraining models from scratch when new data arrives.

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

[1] Zhai et al. 2021. LiT: Zero-Shot Transfer with Locked-image text Tuning

[2] Wallingford et al. 2023. Fluid: A Unified Evaluation Framework for Flexible Sequential Data

We thank the reviewer for their engagement in the rebuttal process. Since the discussion period is coming to an end today, we wanted to follow up on our response from yesterday. While we understand that it has been only several hours since our last response, we wanted to ask if you have any major additional questions or concerns left, and if not, we hope you might consider raising your score.

W3) Serious Design Flaws in Continual Learning Setup

• 4-7 timesteps are too few … major findings would significantly change when tested on 20/50 timesteps

We acknowledge the reviewer's perspective that extending our analysis to 20/50 timesteps could yield different findings. However, we contend that our chosen 4-7 timestep range (one model a year for up to 7 years) is reflective of certain real-world scenarios, particularly in fields where data is subject to rigorous quality filterings and models are subject to rigorous evaluations between deployments. Such models are expected to adapt to slow changes in the data distribution (year to year) and tolerate faster but small changes (month to month). Examples of such settings include image classification for autonomous driving, where updates are thoroughly tested in controlled environments to ensure safety and compliance with regulations. Even models, like ChatGPT undergo updates at a relatively smaller frequencies (After its initial release about an year ago, GPT-4 was only recently updated with new knowledge).

Moreover, our work is the first benchmark for continual training at the scale of 12.7 B image-text pairs (e.g., CLOC [Cai et al. 2021] has only 39M samples, see others in Table 4). This initial exploration into the realm of 4-7 timesteps multi-year web-scale data can pave the way for future work to explore the more challenging settings with 50-60 time steps. We have prioritized the large-scale aspect of our benchmark and tapped the largest sources of publicly available image-text data. Our benchmarks effectively span 14 years from 2008-2022. As such, creating a benchmark with more than 14 years with data prior to 2008 might be infeasible as of now as the scale and quality of earlier data is much less.

Alternatively, our benchmark itself provides a testbed to experiment with finer granularity of months (Sec 2.3). Our dataset release will include time stamped data at the granularity of months. We believe that it would be an interesting direction for future work to explore continual learning at finer granularities in our TiC-DataComp benchmark where we have 70-80 time steps. In such regimes, we imagine that one can explore techniques that alternate between full model updates (which can be done less frequently) and finetuning a few parameters to adapt to more rapid distribution shifts (which can be done more frequently).

• Is memory constrained? For TiC-RedCaps and TiC-YFCC15M, all and exp/equal only differ at last timestep … This issue is mitigated in TiC-DataComp as at last timestep … note: Am I correct that buffer conserved here is 2.3x smaller, not the claimed 3.5x smaller? …

We apologize for confusion here. Table 1 in the draft depicted the correct sizes and there was a minor typo in the text for Cumulative method description in Sec 3. We have updated the draft and reflected the change. In our experiments, we only replay D size of old data, and thus even for TiC-RedCaps and TiC-YFCC15M, we have differences in the last two steps and for TiC-DataComp, we have differences in the last five steps. This also addresses the confusion between 3.5x reduction in memory buffer (3.5x is the correct reduction).

• Replay buffers not utilized effectively, computation implicitly constrains memory. I worry most of the decline in performance is due to random sampling, and hence wanted to see supporting evidence for W2.c).

We have added evidence to the W2c weakness above. We also thank the reviewer for their detailed suggestion on methods to try and for their astute feedback.

The main goal of our work is to construct the continual learning benchmark and evaluate simple but important baselines. We believe that the method suggested by the reviewer is a very interesting direction for future work on how to improve sampling datasets when selecting the replay buffer for continual learning at scale.

W4) Little discussion of past work despite similar findings/parallels

We thank the reviewer for providing references to these papers.

• (Cai et al., 2021) …l aspects of interest here, learning rate modulation, comparing impacts of replay buffer, web-scale training … but on 600K timesteps. ... CL methods work only on a short scale and degrade beyond 5-20 timesteps.

Yes, we did reference the work of Cai et al. 2021 in our draft and have now expanded (in App A.2) our discussion on it in the revised draft. We agree that this work provides interesting discussion/analysis for continual learning at a large number of steps. However, our study differs from Cai et al. 2021, in several crucial respects:

(i) Training Methodology: We employ noisy supervision using contrastive loss between image-text pairs, as opposed to the cross-entropy loss used by Cai et al. 2021.

(ii) Scale of Experiments: Our experiments on the TiC-DataComp dataset are orders of magnitude larger, scaling up by 200x. These differences introduce unique challenges. The use of contrastive loss (i) necessitates a tailored approach to designing our evaluation studies. The significantly larger scale of our experiments (ii) poses challenges in collecting timestamped data and understanding if and how distribution shifts impact learning at this scale. As mentioned before, we believe that extending the study on our TiC-DataComp benchmark (with 12.7B examples) to a finer granularity as in Cai et al. 2021 is an interesting direction for future work.

• Finding (1) on Imagenet IID … in several works, e.g. “Computationally Budgeted Continual Learning: What Does Matter?” … Other works like “One Pass ImageNet” … have analysis on Sequential/Cumulative variants on Imagenet but … not discussed.

We are grateful to the reviewer for suggesting these additional references, which we have now incorporated the suggested papers into our discussion (in App D.1). If the reviewer suggests, we are open to moving the Imagenet results fully to the appendix.

Our Imagenet experiments were primarily inspired by the "loss of plasticity" phenomenon described in Ash and Adams 2020. Their study demonstrates that models sequentially trained on two splits of CIFAR-10 data (initially on 50%, followed by 100% of data) exhibit poorer generalization compared to models trained from scratch on the entire dataset. Since we do not observe this behavior for continual training of CLIP, we investigated the existence of such behaviors on up to 8 splits of Imagenet. Our findings reveal that the simple cumulative baseline remains competitively close to the Oracle model (that benefits from using the full compute budget on the entire pooled training data from the beginning).

Although the referenced works and ours both underscore the effectiveness of continual training on synthetic continual learning setups derived from ImageNet, the specific settings in these studies vary significantly from ours, limiting direct comparisons. A key distinction of our work is the comparison of our final models with an Oracle model, a step not undertaken in the referenced papers as it was not their primary objective.

• Other works which have interesting investigations with computational limits (Bornschein et al., 2022) and (Jang et al., 2022) have nice analysis quite relevant to this work. E.g. hyperparamter optimization as a cost in the compute budget C (will be required when training in the unknown)

Yes, we indeed cite these references in our work and now we have expanded the discussion with these papers. Our methodology maintained consistent base hyperparameters across all timesteps, with the exception of specific ablations on learning rate warm-up and maximum learning rate at a medium scale. Consequently, our experimental framework does not account for the costs associated with hyperparameter selection.

What is the size of dynamic retrieval and LAIONNet test sets per chunk?

We have now added sizes of our evaluation tasks in App C.3.

The backward and forward transfer metric utilized … from (Lin et. al., 2021). (Lopez-Paz & Ranzato, 2017) have a … different metric for backward and forward transfer, while (Díaz-Rodríguez et al., 2018) do not … modification to this metric.

Thanks for highlighting the differences; we have clarified this in the updated draft (in App E). The main commonality in all of our works is the same performance matrix $\mathcal{E}$. While Díaz-Rodríguez et al., 2018 used the same forward transfer metric as ours, the backward transfer metric in their work and metrics in Lopez-Paz & Ranzato, 2017 performed centering by subtracting the ID performance. Note that, as discussed above, we have now updated the metrics to include the metrics used in Díaz-Rodríguez et al., 2018 in our work.

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

• To what degree is the drop in forward transfer attributable to tasks getting harder vis-a-vis encountering new distributions.

This is a great question. To elucidate the differences, we compared performance of the Oracle method trained on data from all time steps on evaluation sets from different time steps, i.e., The last row in dynamic evaluation of Oracle method in Figure 4 (right) and in Figure 6 (App B).

Since the models evaluated in the last row of Oracle method are trained on all training data, differences in performance of the Oracle model at the last training time step primarily stem from hardness of distributions at every time step.

We make the following observations. For TiC-YFCC and TiC-Redcaps, we observe an increasing trend in dynamic retrieval performance highlighting that the task becomes easier for future time steps. For TiC-Datacomp, the hardness of the task remains almost the same (slight increase) as the retrieval performance remains almost the same. These observations hint that while we encounter different distributions for all datasets, for TiC-Datacomp the hardness of the tasks in future timesteps remains almost the same, whereas, for TiC-YFCC and TiC-Redcaps, the tasks become easier as time progresses. This observation hints that for TiC-YFCC and TiC-Redcaps, the quality of images and their corresponding captions improve over time. For example, the average quality of cameras on mobile phones have improved.

W2) Major results simply not present

We have now included all the missing tables in the appendix or references to already existing tables in the appendices.

• Same maximum LR works best across all runs when using cosine schedule

Table 7 in the updated draft provides evidence to this finding (we have added the missing reference in the main paper). Earlier draft also included this table with a typo in one column – we have fixed that.

• Warm up helps training on data from the first time step, but hurts on subsequent time steps – After correcting the minor shift issue, the table … support the opposite conclusion.

Sorry for the confusion here. We copied the column names incorrectly (you can check the mismatch between column names and the corresponding results between Table 6 and Table 2). We have now fixed this table in the updated draft which matches the stated finding in our paper.

• Given a fixed buffer size for each past step, we observe minimal to no difference between random subsampling and other strategies. – Didn't find supporting evidence

We only performed preliminary analysis for this. In our preliminary experiments, we explored the efficacy of subsampling old data based on the alignment between text and image content from previous time steps. Specifically, when training a model at time step t+1 , we used the model from the end of time step t to assess this alignment. We employed two distinct subsampling methods:

1. Retaining half of the data with the lowest alignment scores, based on the premise that these data points might be more challenging to learn and require additional gradient steps.
2. Retaining half of the data with the highest alignment scores, under the assumption that these represent higher quality data, as indicated by the stronger alignment between text and image pairs.

We applied these methods to the TiC-YFCC dataset and evaluated their performance against a baseline of random sampling. The outcomes revealed minimal differences: less than 0.2% variation in Imagenet performance and under 0.5% in dynamic retrieval performance across different time steps. Given that these minor improvements came with a significant computational cost—requiring a full forward pass to compute alignment post each training epoch—they exceeded our compute budget constraints. As a result, we opted for random sampling in our research. These findings have been included in the appendix for further reference (App B.4). We leave investigation on improved subsampling techniques for future work.

• I cannot find any results for LAIONNet-like dynamic classification task.

We apologize for missing the table for this result. We have included this in the updated draft and have added a reference to that in the main paper. The trends described in Sec 3.4 hold true. We have added Table 8 in App B.6 and Figure 15 in C.6.

We thank the reviewer for their extremely detailed constructive feedback on our work. It is encouraging to note that the reviewer recognizes the high quality of our paper, its relevance in addressing an important problem, and the depth of our analysis.

We provide detailed answers to your concerns below and we believe that by addressing these concerns we have significantly improved the manuscript.

W1) Sequential and cumulative models behave quite differently between TiC-YFCC15M and TiC-Datacomp

We give a brief answer here and provide a details as we answer the sequence of questions below. The primary reasons are two (see Sec B.7 for detailed discussion):

(i) the nature of the distribution shift observed in YFCC and its constructed dynamic retrieval task; we observe that models trained on Tic-YFCC suffer from relatively larger drops due to catastrophic forgetting than Tic-Datacomp (see updated plots for Sequential in Fig 6 in App B.1).

(ii) compute used at each time step per data available at each time step. Overall YFCC is 2x smaller than Tic-Datacomp (M) but the compute we used in both TiC-YFCC and TiC-Datacomp setup is of similar order (in fact, it is slightly higher in TiC-YFCC). We re-ran the experiments for Tic-YFCC by reducing the compute. In the updated runs, we observe that the gap between ID performances of Sequential and Cumulative-All vanishes .

We also emphasize that the goal of our study is to highlight the problem with the distribution shift for data collected from different data sources (e.g., Flickr, Common Crawl, Reddit). While we agree that the distribution shift severity is different in different data sources, we show that the problem exists for all different data sources. Similar to Figure 4 (right), we have now added heatmaps with other methods and datasets on dynamic retrieval evaluation tasks (Figure 6). For a fixed training time (a specific row), we clearly see performance drop on evaluation time steps after the training time step, highlight distribution shift in these benchmarks.

• … does TiC-DataComp have significant distribution shift

Yes, we observed that TiC-Datacomp has distribution shift problems as shown by the performance difference between forward transfer and backward transfer (in Table 2, Figure 4 (right) and Figure 6 (c,d)). We clearly observe that performance on dynamic tasks from future time steps is worse than current or previous time steps. Figure 3 presents some examples highlighting how distribution evolves over time in TiC-Datacomp. For example, with time images from novel concepts appear (e.g., COVID-19).

• Can the authors create a plot similar to Figure 15 for TiC-DataComp or some other mechanism to analyze distribution shift in TiC-Datacomp?

We have included a plot highlighting the nature of the shift in TiC-Datacomp in updated Figure 16. We observe similar trends for FID score on feature distribution for TIc-Datacomp. For TV distance, the distance increases initially and then saturates for later timesteps.

• In TiC-YFCC15M and TiC-RedCaps, … forward transfer is significantly affected, but … not … in TiC-Datacomp. Similarly, gap between backward transfer and ID retrieval is far smaller between them in TiC-Datacomp. Why is this the case?

We primarily attribute this to the data source differences between these constructed benchmarks. While we agree that the distribution shift severity is higher in TiC-YFCC and TiC-Redcaps, we show that this problem exists even for TiC-Datacomp. We have now included Figure 6 where we plot the performance evaluation matrix $\mathcal{E}$ for all of these datasets.

• Fig1 (mid) and Figure 10 present a different picture than Table 2. … In Table 2, intuitively, ID performance should be higher (equal)

Your observation about the discrepancy in the forward and backward transfer metrics is insightful. Thank you for highlighting this. Originally, our methodology involved averaging the entries in the upper and lower diagonal of our performance matrix $\mathcal{E}. This approach inadvertently results in the backward transfer metric being influenced by later evaluation time steps resulting in backward transfer performance numbers slightly larger than ID performance.

To address this issue, we've revised our metric calculation method by deviating from prior work Lin et al 2021. Now, we normalize the data in each row by subtracting the ID performance as in Díaz-Rodríguez et al., 2018) This adjustment ensures a more balanced and accurate representation across all training time steps. We have also adjusted the forward transfer metric similarly (Table in Appendix ). We have updated the draft with these change (App E). In the final version, we will include these results in addition to the existing results.

We have also included additional plots that capture the performance matrix \mathcal{E} as in Figure 4 (right) on all datasets in the appendix (Figure 6 in App B.1).

All replies except the ImageNet-IID claim seem satisfactory to me (the (Díaz-Rodríguez et al., 2018) part particularly was quite neat, thanks for this!). The ImageNet-IID claims seems to misrepresent OPIN and CB, as I explain below.

Our findings reveal that the simple cumulative baseline remains competitively close to the Oracle model (that benefits from using the full compute budget on the entire pooled training data from the beginning)... the specific settings in these studies vary significantly from ours, limiting direct comparisons. A key distinction of our work is the comparison of our final models with an Oracle model, a step not undertaken in the referenced papers as it was not their primary objective.

Both works compare with an Oracle model!

[OPIN] As per my understanding, the main comparison in OPIN is between Oracle and continually trained models eq to sequential and cumulative-prioritized replay, it seems misleading to claim the contrary. Precisely, Table 1 compares performance at the last timestep between -- Multi-epoch (90 epochs) is Oracle as it does full passes over the data multiple times, One-Pass (Naive) is sequential and One-Pass (Prioritized Replay) is a variant equivalent to Cumulative-prioritized replay (similar variety to equal/exp). Table 2 extends to Oracles with different compute budgets.

[CB] I am less confident about this, but to the best of my knowledge the equivalence should be as follows -- ERM-Naive is Oracle, Naive is the same as cumulative-all. Minor note: This was also the citation for the issues in expensive sampling strategies point I made.

Overall, both works make the same conclusion made here as far as I see on Imagenet-IID, but have far more extensive analysis (the conclusion is not a new finding). I don't see what does the ImageNet experiment say more than preliminarily confirming the conclusion presented in OPIN (and CB for the increased split case). Both works are more recent than (Ash and Adams, 2020) and specifically have ImageNet-IID experiments same as in this work.

W3.1) [Too few timesteps] I am not convinced on this and would push back. Setting this norm would be important for a benchmark paper, as a lot of methods in continual learning fail to mitigate forgetting beyond a few timesteps. Similarly, I am concerned there will be a far larger gap between CL methods and -all, and -all and Oracle. However, with the correction in buffer being D and not 2D this is no longer a critical concern considered for score of the work, just an drawback of this benchmark. If this is specifically acknowledged it would be sufficient for me.

W3.2) [Is memory constrained?] Resolved fully, thanks!

[W3.3) Replay buffers not utilized effectively] This has not been resolved at all, and is a critical drawback. The specific issue pointed out is data-imbalance and using unique samples. The sampling strategy this implies testing with is to balance the number of samples selected between past/current data and prefer deleting the samples which were used for replay over samples which were not when buffer size is constrained. I would strongly recommend testing this correction and presenting results as this is critical for the difference between cumulative-eq and cumulative-all for Datacomp-medium, and in the final version for all datasets.

The authors instead explored unrelated sampling strategy which had little to do with the question and requires additional forward passes (similar strategies have been described to not work in past literature precisely due to being too expensive).

W3.2 and W3.3) were the primary concerns I had in this work. W3.2 and W1 were addressed, and I shall raise the score to 6 to reflect this. If W3.3 is sufficiently addressed on the smallest Datacomp dataset, I would have no further concerns and can commit to raising my score to 8.

W1) Sequential and cumulative models behave quite differently between TiC-YFCC15M and TiC-Datacomp

Authors main visualization support is: a) We have now included Figure 6 where we plot the performance evaluation matrix for all of these datasets.

with conclusions: (i) Tic-YFCC suffer from relatively larger drops due to catastrophic forgetting than Tic-Datacomp (ii) On re-running the experiments for Tic-YFCC with reduced compute, the gap between ID performances of Sequential and Cumulative-All vanishes.

The authors additionally provided:

(i) We have included a plot highlighting the nature of the shift in TiC-Datacomp in updated Figure 16. We observe similar trends for FID score on feature distribution for TIc-Datacomp. For TV distance, the distance increases initially and then saturates for later timesteps.

On this point, the updated TV-distance plots did not have increasing drift. To ensure that the drift is not simply between the first timestep and the rest, can the authors extend the Datacomp plot the same graph across some years in the middle and the end timestep additionally?

I am very happy/convinced by the rebuttal on this point and tentatively consider this fully resolved. I would be very happy if I could see the Figure 16 plots across 2-3 timesteps before rebuttal deadline to complete this point.

Hi!

Sorry, maybe that point was too harshly put. The sampling structure that I was thinking about:

[Memory storage allocation, not a)]: Say in -equal strategy, we would storing D/2 examples from each timestep, and the next iteration D/3 from each past timestep -- here the degree of freedom is how can we choose D/3 out of D/2 samples. My suggestion is prioritize keeping the samples in D/3 which were not/less used for training previously, and break ties by choosing randomly.

To clarify, samples in the D/3 will be subsampled from D/2, not being disjoint. I presume the standard of one loses access to all samples once discarded, yes!

Now given the buffer, how to choose samples for training-

[Sample selection from buffer, use b)] Given that we have fixed the buffer, we know that we have D samples of current timestep and say D/3 samples of each of the past timesteps, we would over-sample D/3 samples compared to current samples, yes. This is avoiding the data imbalance problem. I am relying that simple augmentations of the past samples like randomflip while oversampling would mitigate overfitting to a greater extent.

So, not (a) but the sampling I mentioned above which is slightly different from random and (b) I wanted to note augmentation as it might help?

Overall, just wanted to give a strategy which I think mitigates this issue cheaply but I expect to be far better than random sampling, trying to maximize performance. If the authors find clever tweaks (I only spent a few minutes atbest thinking about this at the best), please feel free to make them. I would highlight choosing the oversampling factor was important-- I've found that oversampling by a square root of the proportion worked the best (i.e. if ratio is D/6 old samples in a given timestep to D samples of latest timestep, the sampling is √6:1 between them) but maybe authors find something more principled/better working.

We thank the reviewer for their elaborate response and for clarifying the suggestion made for W3.3 about using the count and over-sampling while replaying old data. We ran additional experiments and discuss our results below:

W1) Sequential and cumulative models behave quite differently … very happy/convinced by the rebuttal on this point … would be very happy if I could see the Figure 16 plots across 2-3 timesteps before rebuttal deadline to complete this point.

We thank the reviewer for appreciating our responses on W1. As per your suggestion, we have updated Fig 17 by including heatmaps for TV distances between various reference timesteps, not just 2016. In this revised figure, we can see that as we move away from the reference timestep, the TV distance increases for all reference time steps (represented by different columns). Additionally, we noticed that the TV distance magnitude decreases relatively when later time steps are used as references. We believe that this is due to the inclusion of data from previous years within the 2016 dataset.

Tasks getting harder over time … Would like to have the bestpool results "For TIC-DataComp (XL), we include results with Bestpool filtering." together in this table or alternatively with and without bestpool filtering in Table 5.

We have updated Table 5 with your suggestion. For the final draft, we will consider integrating this result with Table 2.

[W3.3) Replay buffers not utilized effectively]. The sampling structure that I was thinking about … [Memory storage allocation, not a)] … [Sample selection from buffer, use b)] … Overall, just wanted to give a strategy which I think mitigates this issue cheaply but I expect to be far better than random sampling, trying to maximize performance.

We have now included results with the suggested method in Table 18 (in Appendix E.2). As the reviewer suggested, we have made the following two modifications: (i) Instead of random sampling, we implemented the count based subsampling that prioritizes not/less used examples; (ii) We oversampled data from old timesteps with ratio inversely proportional to the ratio of examples, i.e., if the old data is of size D/2 and the new data is of size D, then we upsample old data with 2:1 ratio.

However, we observe that this method doesn’t improve performance over Cumulative-equal (and in fact hurts the performance). We hypothesize that this can be due to the decreasing marginal utility of labeled data as highlighted in existing work [1]. Their work argues that due to information overlap among data, as the number of samples increases, the marginal benefit a model can extract from the data diminishes. As a result, Cui et al. [1] proposed using of “effective sample size” instead of actual number of samples to obtain the ratio used to perform re-sampling or re-weighting. We include the expression of effective sample size in more detail in Appendix E.2.

In our settings (even at small scales), our datasets contain an order of 100k img-text pair even after subsampling data from old timestep. For example, with -equal baseline, when training on the last timestep (i.e., 2022), the smallest dataset (i.e., 2016) is of approximately 400k samples. Plugging in the expression for effective sample size from Cui et al. [1], we observe that for all $\beta \in (0, 0.99999)$, the ratio of effective sample sizes for different timesteps remains close to 1. This may highlight why our naive over-sampling strategy doesn’t improve over no-oversampling.

Due to a shortage of time and compute, we didn’t perform ablations with $\beta$'s arbitrarily close to 1, but if the reviewer suggests we would be happy to run those experiments for the final version since we have already implemented the oversampling code. We will explore the benefits of count-based subsampling instead of random sampling separately from oversampling in the final version.

[1] Cui et al. Class-Balanced Loss Based on Effective Number of Samples. CVPR 2019.

[W3.3] Interesting, oversampling does not help. Performance degrades due to overfitting. This resolves all my main concerns, I raise my score to 8.

[W Part 2] Yep, resolved! Thanks. I agree that many continual learning setups have slightly different assumptions from each other, preventing direct comparisons.

We thank the reviewer for their feedback and for the positive assessment of our work.

The dataset being solely focused on training CLIP may be somewhat limited. Can the article consider incorporating more vision-language models?

While our paper primarily focuses on experiments with training CLIP models, the constructed benchmark with timestamped image-text pairs can be used to train other Vision Language Models (VLMs). Moreover, since, CLIP vision encoder and text encoder are the basic blocks for other VLMs, e.g., Stable Diffusion models [1] and LLava models [2], we believe that improving CLIP models have direct implications for other VLMs. We would be happy to run additional experiments if the reviewer has any specific suggestions.

The YFCC100M dataset might be somewhat outdated in terms of the years it covers. It may be more representative to explore newer datasets for the research.

We agree with the reviewer that YFCC100M contains data from timestamps before 2014. However, we want to highlight that we also collected Tic-Datacomp which contains 12.7 B data points till timestamp 2022, overall giving us data from 2004–2022.

The fundamental issue … catastrophic forgetting … fine-tune a small number of parameters (e.g., prompt tuning) … catastrophic forgetting a major concern? … if we fine-tune a large number of parameters, resource limitations may become a factor … is it necessary to construct such benchmarks?

In the majority of CL benchmarks, catastrophic forgetting can be attributed to significant and abrupt changes between consecutive timesteps, often across the data distribution, the targets, and the task. For example, the data and targets in perm-MNIST or split-MNIST have little relation to each other from one task to another. Our setup of continual pretraining for foundation models differs from these existing CL benchmarks in two crucial ways: (i) we observe little catastrophic forgetting; and (ii) by observing new data we not only benefit on tasks from current time step but also improve performance on tasks from old time steps (these observations are supported with Fig 6 and Fig 4 (right)).

At a high level, by continually training on more data, we hope to improve “general-purpose” abilities of foundation models (e.g., performance on Imagenet). Thus while fine-tuning a limited set of parameters may be an effective strategy for tailoring a model to a confined dataset from a specific domain, this approach is less viable for maintaining and updating general-purpose foundational models with the dynamically evolving internet data. Given the proportional increase in new data relative to the pre-existing dataset, it necessitates comprehensive model updates.

(New result) As per your suggestion, we performed an additional experiment to assess the efficacy of updating selected model parameters versus full-scale fine-tuning with sequential training on the TiC-YFCC dataset. In particular, we compare LiT [1] where we perform full-finetuning on the first time step and then perform LiT for subsequent time steps on Tic-Datacomp (M). We observed that while updating a few parameters does avoid catastrophic forgetting on old timestamps, this happens at the cost of reduced performance on all tasks. For instance, the final Imagenet performance drops substantially from 24.0 (w full finetuning) to 7.3 (w LiT). This highlights limitations of finetuning only a small number of parameters while trying to keep general purpose foundation models up to date.

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

[1] Zhai et al. 2021. LiT: Zero-Shot Transfer with Locked-image text Tuning