FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources

We sincerely thank the reviewer for their valuable review.

Q1: Data parallelism mentioned in this paper does not show promising results compared to OpenCLIP (in Figure 3). Is there any other significant difference between OpenCLIP and FastCLIP?

A1: We would like to note that we did not propose a new parallelism paradigm in this paper. Both OpenCLIP and FastCLIP leverages data parallelism. Instead, we proposed a novel gradient reduction strategy that eliminates part of the gradient communication. At 8 nodes, FastCLIP has a 1% to 3% reduction in terms of training time compared with OpenCLIP (please refer to Tables 15 to 22 in Appendix E, which present the per-iteration training time of FastCLIP and OpenCLIP on different scales and in different environments). Besides the gradient reduction strategy, another key difference between OpenCLIP and FastCLIP lies in the loss function, where OpenCLIP minimizes the Mini-Batch Contrastive Loss (MBCL) while FastCLIP optimizes the Global Contrastive Loss (GCL) family. Compare to OpenCLIP with MBCL, FastCLIP with GCL does not require a large batch size to obtain good performance, thus enabling researchers to train performant CLIP models in limited-resource settings.

Q2: Figure 4 (a) does not show convergence on the accuracy curve which could be problematic. Should the training accuracy converge?

A2: To our knowledge, large-scale ViT-based CLIP models requires a large amount of samples seen before convergence, which is not affordable even for researchers with a large number of resources. For example, in the official repository of Cherti el al. (2023): https://github.com/LAION-AI/scaling-laws-openclip/blob/master/figures.ipynb, from the last figure we can see that there are not clear signs of convergence after 34B samples seen. Thus in the literature, the common practice is to train the model for a given number of samples seen (e.g., 12.8B) to obtain a decent model. Curves similar to those in Figure 4 (a) in our paper can also be found in other works, e.g. Figure 1 in [1] and the figures in [2].

[1] Hu et. al. Demystifying CLIP Data. ICLR 2024. https://openreview.net/forum?id=5BCFlnfE1g

[2] open_clip. https://github.com/mlfoundations/open_clip/blob/main/docs/PRETRAINED.md

Q3: Figure 4 (b, c) do not show a significant difference in scalability.

A3: We would like to note that FastCLIP and OpenCLIP shares similar trend in Figure 4 (b, c). Moreover, FastCLIP only requires few resources (GPUs) to obtain good performance while OpenCLIP requires a large amount of them. As the curve in Figure 4 (b, c) predicts, the larger the number of nodes is, the worse the efficiency will be. This indicates that FastCLIP is at an advantage over OpenCLIP in terms of efficiency.

Q4: The experiment part needs more details. What do the numbers represent in Table 3, 4, 5, and Figure 2?

A4: We provided a brief introduction to the metrics we use in Section 5 (Line 347). To be specific, IN & Variants denotes the average top 1 accuracy on ImageNet-1K and 6 ImageNet distribution shift datasets; Retrieval denotes the average of mean recall at 1 on Flickr30K, MSCOCO and jaccard score on WinoGAViL; and Datacomp denotes the average performance on 38 tasks, which is a combination of accuracy on classification tasks and recall/ jaccard score on the three Retrieval tasks, as documented in Gadre et al. (2023). The numbers in the parentheses denote the standard deviation over 3 runs. Improvement denotes the absolute performance difference between FastCLIP-v3 and OpenCLIP on different metrics. We have revised our manuscript to include more detail about the metrics at Line 347 and at the captions of the tables and figure.

Q5: FastCLIP is built upon existing techniques and is not novel.

A5: We refer the reviewer to Q1 in the general response for explanation.

Q6: The writing style needs improvement, and some part of this paper needs to be summarized.

A6: We refer the reviewer to Q2 in the general response for explanation.

We sincerely thank the reviewer for their valuable review.

Q1: The authors did not conduct extensive ablation study on large scale datasets due to resource constraints, which may limit the generalizability of results on extremely large data scales. Would additional experiments on extremely large datasets (>1 billion examples) align with the current findings for smaller datasets?

A1: We are conducting an experiment of FastCLIP-v3 on a 1.4B subset of the DFN-2B dataset (Fang et al. 2023). The number of samples finished at this moment is 11.2B. We compare the performance of FastCLIP-v3 on this data to that trained on LAION-315M reported in the paper in the following table, where the ImageNet-1K top1 accuracy at different number of samples seen are presented. We can see that our approach is still effective on extremely large-scale data. We also plot the performance curves of FastCLIP-v3 on different datasets and present it in Appendix E.2 of the revision. We expect to finish the training with 12.8B samples seen and add the result in the next revision.

Q2: FastCLIP still requires access multiple GPUs, which may limit applicability for highly resource-constrained users.

A2: We would like to note that compared with OpenCLIP which requires tens of to hundreds of GPUs, FastCLIP already makes a big step at reducing the number of GPUs for large-scale CLIP training (8 in our experiments). In our paper, we experimented with hundred millions of data. Hence, in order to finish our extensive experiments we use 8 to 16 GPUs. We believe this level resources can be easily available to many researchers in the community. It is an interesting direction to further lower down the resource requirement to accommodate the need of highly resource-constrained users.

Q3: How does the performance of FastCLIP vary across different types of GPUs or compute environments?

A3: In terms of downstream performance (e.g. zero-shot classification and retrieval), the results are not dependent on types of GPUs or compute environments. In terms of training time, we profile the results on different clusters with different types of GPUs and interconnect, which are presented in Section 6 and Appendix E.2. FastCLIP-v3 demonstrates consistent improvement over OpenCLIP under different settings.

Q4: Could other advanced optimization techniques (e.g., adaptive optimizers beyond AdamW) further enhance the framework's efficiency?

A4: In this work we tested 4 optimizers on different scales and found that AdamW achieves the best overall performance. There is possibility that other optimiers outperform AdamW for CLIP training. It is an important research question and we leave it for future work.

Q5: The gradient reduction strategy and optimization techniques lack novelty.

A5: We refer the reviewer to Q1 in the general response for explanation.

We sincerely thank the reviewer for their valuable review.

Q1: The applicability of FastCLIP is limited due to the assumptions made in [1].

A1: We respectfully disagree with the reviewer. [1] has demonstrated its effectiveness of using a small batch size compared with previous approaches, e.g., SimCLR. This optimization technique has been demonstrated to be effective across a broad range of problems (e.g., LibAUC [2]). The assumptions made in [1] are fairly standard for analyzing the convergence of optimization algorithms. We noted that most existing papers of CLIP training may focus on other perspectives but simply use existing optimization algorithms e.g., Adam. However, the convergence analysis for any optimization algorithms would require some assumptions of the problem, and this does not prevent them from being applied in practice. Similarly, the effectiveness of our framework FastCLIP has been demonstrated compared with OpenCLIP with a small batch size through extensive experiments across various compute and data scales.

[1] Yuan Z, Wu Y, Qiu Z H, et al. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. ICML 2022.

[2] Yuan Z, Zhu D, Qiu Z H, et al. LibAUC: A Deep Learning Library for X-risk Optimization. KDD 2023.

Q2: OpenCLIP is designed for large-scale clusters and large batch sizes, but the authors compared FastCLIP and OpenCLIP in only small batch settings, violating the original conditions under which OpenCLIP was evaluated.

A2: OpenCLIP is the state-of-the-art distributed training framework for CLIP training. We addressed a different scientific question "how to train a CLIP model on billion-scale data with limited compute resources?" Although this is not a main-stream approach as industry leads the research of CLIP training with large compute resource, we believe it is a fundamental problem that requires rigorous research. To the best of our knowledge, there are no existing distributed training algorithms of CLIP training that work well with a small batch size. Hence, our work would pave the way for latter works on accelerating CLIP training with limited compute resources. We would like to point out that results in Table 6 showed that FastCLIP with small batch size already has comparable performance to OpenCLIP with large batch size.

Q3: The results in Table 4 and 5 are inconsistent, with different methods showing unstable performance.

A3: We would like to note that Table 4 shows the results of FastCLIP-v0 to -v3 with the AdamW optimizer, while Table 5 shows the results of FastCLIP-v3 with different optimizers. Thus the 4th and 8th rows of both tables overlap with each other, which are identical and consistent. In Table 4, FastCLIP-v3 achieves 1st place in 4 of the 6 metrics (3 in the medium-scale setting and 3 in the large-scale setting), and 2nd place in the remaining 2 metrics. Thus we arrive at the conclusion that FastCLIP-v3 is the overall most-performant method for CLIP training in the Global Contrastive Learning framework. The same reasoning applies to the AdamW optimizer in Table 5.

Q4: FastCLIP is built upon existing techniques and is not novel.

A4: We refer the reviewer to Q1 in the general response for explanation.

Q5: The writing of this paper lacks clarity.

A5: We refer the reviewer to Q2 in the general response for explanation.