Vision-Language Dataset Distillation

Summary:

Thank you for the insightful comments and questions. We appreciate your recognition of the clarity of our writing, the compelling experimental setup. We address the raised concerns below.

W1. Weak baselines and no theoretical analysis.

First, with regard to theoretical analysis, we would like to clarify that dataset distillation is inherently an empirical field, at least currently: a number of methods have been proposed for distilling image classification datasets (Cazenavette et al., 2022, Cui et al., 2022, Deng & Russakovsky, 2022) (published in CVPR, ICML and NeurIPS respectively), all of which rely exclusively on empirical analysis and evaluation. While theoretical grounding would certainly be beneficial to the dataset distillation field, it is outside the scope of our paper. As we note in the abstract, our core contribution is “proposing the first Vision-language Dataset Distillation method” by “implicitly matching the long-range training bi-trajectory of the target vision-language data and the synthetic ones” given three key challenges mentioned in Sec. 1, and verifying the strength of the method empirically.

Second, about coreset baselines, our work extends these methods to vision-language datasets, which introduces new challenges and complexities. The coreset methods we compare against are among the most effective and widely used baselines in dataset distillation [a, Cazenavette et al., CVPR, 2022, Wang et al., CVPR, 2022, Zhao & Bilen, ICLR, 2021b]. Since there are no prior methods for dataset distillation on vision-language datasets, leveraging coreset selection methods allows for a fair comparison between our synthetic distilled dataset and a real dataset subsampled to fit under the same storage budget. In addition, we perform ablation studies (Sec. 4.4) to examine the effects of different parts of our method. We would welcome suggestions for additional baselines or ablation studies. Unfortunately, as mentioned above, theoretical justification is outside the scope of our work – and theoretical analysis by itself would be a substantial contribution to dataset distillation, even on the original image classification task.

W2. The usage of cosine similarity.

We use cosine similarity because it is well-suited for measuring the alignment of high-dimensional data in embedding spaces. In contrastive learning, the goal is to ensure that similar sample pairs are close in the embedding space, and the dissimilar pairs are far apart. As stated in [b], the cosine loss induces sixth-order dynamics (while the L2 loss induces a third-order one), in which a stable equilibrium dynamically emerges even if there are only collapsed solutions with given initial parameters. Cosine similarity has been widely used in representation learning with various theoretical interpretations[c, d]. We refer the reviewer to those works for further details.

W3. The choice of freezing encoders.

Our decision to freeze the text encoder backbone but not the image encoder follows common design choices in the field. This choice is often made to balance computational efficiency with model performance, especially considering the complexity and diversity of visual data compared to textual data. In Tsimpoukelli et al. [e] they only finetune the vision encoder (NFNet) and generate embeddings that are fed into a frozen GPT-3. MEGMA[f] also used NFNet as the vision encoder and a frozen language model GPT-Neo. Scialom et al.[g] showed a learned linear transformation is sufficient for BERT to encode image region representations. Our design choice aligns with the established method in the field, where the image encoder is fine-tuned to better handle the complexity of visual data while the text encoder is often fixed to maintain the computational efficiency and leverage existing language understanding.

W4. Why retrieval? What about distilling the subset of CLIP?

This is an excellent question about using a subset of CLIP for the experiment, and we had considered it. However, even scaling up from image classification to the (arguably simplest) vision-language task of retrieval posed significant challenges (as shown in Tab. 2 and Tab. 5). As we were already tackling an ambitious goal of developing a novel dataset distillation method for vision-language datasets, we decided to keep the evaluation task for now as simple as possible. Additionally, even the subset of CLIP training set could reach a scale of millions (e.g. Conceptual Captions 3M), which demands significant computational resources. Our choice of task and dataset was driven by the desire to introduce and validate our novel dataset distillation method in a context that is both challenging and feasible within our resource constraints.

Thank you for your insightful review. We appreciate your acknowledgment of the clarity and effectiveness of our method and the significant improvements demonstrated in our results. We address the raised concerns below.

Weakness: Computational Demands.

Regarding the computational demands, the reviewer correctly identified the heavy resources required for trajectory data storage and multimodal distillation. As mentioned in Sec 4.2, our models, trained at 224 × 224 resolution on a single RTX 3090 GPU with 24GB, require approximately 40 minutes per epoch for 10 epochs. Distillation takes between 6 to 15 GPU hours, depending on factors such as the number of distilled pairs, using an 8-GPU A6000 node. This demonstrates the feasibility of our approach in similar research environments. In comparison, dataset distillation for image classification tasks typically demands less computation, due to lower data resolutions (e.g., CIFAR10/100: 32x32 [Cazenavette et al., 2022], ImageNet1K & Tiny-ImageNet: 64x64 [Cazenavette et al., 2022, Cui et al., 2022], ImageNet-subset: 128x128 [Cazenavette et al., 2022]). Distillation times for CIFAR10/100 range from 83 to 317 minutes, and for Tiny ImageNet, from 183 to 433 minutes. [Cazenavette et al., 2022]. Note that our work was conducted in an academic lab with resources that, while substantial, is not on par with industry-scale clusters.

Question: Distilling Text at the Token Level.

Our method distills at the sentence embedding level and is primarily designed to rely on the association between entire images and complete sentences for image-text retrieval tasks, where distilling at the token level may not effectively serve our purpose. While operating at the token level is feasible, it would significantly increase computational costs and more storage, which contradicts the principle of dataset distillation aimed at compact representation. Therefore, our decision to use sentence embeddings is more efficient and aligned with the specific needs of image-text retrieval tasks. We opted to limit our scope to sentence-level distillation for this work, considering both storage efficiency and the specific requirements of image-text retrieval tasks.

We hope this addresses your concerns and look forward to any further feedback!

Summary:

We appreciate the reviewer's insightful feedback and acknowledgment of our contribution as the first work to explore vision-language dataset distillation. The recognition of our experimental setup and the significant performance improvements our approach achieves are highly encouraging. We address the concerns raised below.

(W1) Method.

Our approach extends the MTT formulation to handle the unique requirements of multimodal tasks, reflecting a significant departure from standard dataset distillation approaches that focus solely on single-modality data (Sec. 3.1). Unlike previous dataset distillation works, vision-language datasets lack a discrete set of classes (Sec. 1). Additionally, the cross-modal relationships between visual and textual elements require a co-distillation approach (Sec. 4.4). The complexity of those high resolution datasets, combined with more complex models, present significant challenges (Sec.1 & 4.2).

(W2) Text Representation.

Text embeddings offer a compact representation that aligns with the objectives of dataset distillation. They efficiently capture essential information from textual data in a condensed format and are a widely accepted choice in the NLP field [a, b, c, Radford et al., 2021]. This also avoids issues related to gradients, variable length, serving as a strong and standard way to benchmark the task. Many distillation methods' dependence on gradients lead to issues such as vanishing or exploding gradients. Leveraging stable representations of text embeddings bypasses the more challenging process of optimizing via discrete text distillation. Our sentence-level text embeddings provide a uniform, fixed-length representation regardless of the original text length, maintaining a consistent input size without the need for truncation or padding.

Q1: Formulation of K.

In problem formulation, K is the number of sentences associated with one image in the original dataset, which is a fixed number provided by the datasets. K=5 for both Flickr30K and COCO (as mentioned in Sec. 4.2). For $\hat{y}_i$, we aim to use one (instead of five) sentence per image in the distilled set for a more compact representation to learn the vision-language connection. We modified Sec. 3.1 and clarified this part.

Q2: Symbol Notation.

As mentioned in Eqn. 1, \theta^{*} represents the optimal model parameters obtained after training on the entire dataset, while \hat{\theta} denotes parameters obtained from the distilled dataset. In Eqn. 3: The summation over $y’$ is the sum over all possible $y$ values except the current $y$ in the batch. This is a common notation in contrastive learning, where for a given pair $(x, y)$, you calculate the similarity with every other $y'$ in the batch (where $y' \neq y$) to form negative pairs for the contrastive loss. We modified Sec. 3.1 and Sec. 3.2 to clarify this part.

Q3: Contribution from BERT.

Compared to conventional dataset distillation which focuses on image classification, pretraining BERT on a diverse language corpus allows for a more efficient distillation process, and reduces the size of the distilled dataset. As shown in Appendix Tab.10, it demonstrates the importance of pretraining and without pretraining, BERT trained from scratch performs poorly, even with the entire dataset. Pretraining is a common approach to provide a good foundation and starting point.

The core of vision-language dataset distillation is to figure out the most compact way to learn the joint space of the two domains. While both the encoders are pretrained, they are only pretrained on unimodal data (i.e. ImageNet for image encoder and BooksCorpus and Wikipedia for BERT) and they have not been trained on the multimodal embedding space and had no exposure to the other modality. The distillation process still heavily relies on the interaction between vision-language data.

Q4: Performance Impact of Pretraining Datasets.

Thank you for raising an important point regarding the dependency of models on the pretrained datasets. Given that most models we use, such as NFNet, typically have only one available pretrained version (pretrained on ImageNet), we are conducting an additional experiment to pretrain the NFNet on PASS [d] dataset. Due to the limited time available for the rebuttal period, we may not complete the pretraining within this short time. However, we will provide an update on this in the final version of the paper.

Summary:

Thank you for your insightful review and for acknowledging the contributions of our work, especially the novel setting, the compelling experimental setup and the clear presentation. To address the two noted weaknesses:

(W1) Noticeable Changes in Images.

We have found that increasing the learning rate or distillation time does lead to more noticeable changes in the images within the distilled dataset. We visualized the distilled image at iteration = 7000 and learning rate = 5000 in Fig.4, Appendix Sec. A in the updated paper. However, it's important to note that a higher learning rate or longer distillation time does not necessarily translate to improved performance of the distilled dataset, even if the images appear to deviate more drastically from the human perception perspective. Changes in image pixels alone may not reliably predict distillation performance. It is rather a measurement of the strength of distillation. More distorted images suggest uneven pixel updates, while even updates yield results similar to the visualization we provided before (e.g. Fig. 3).

In line with previous studies, we initially expected more obvious changes in the images would lead to better performance, but our findings suggest a different behavior of in vision-language distillation with trajectory matching framework and it reflects how models capture the vision-language interaction. From a human perception perspective, the distilled images appear to be moving less compared to previous classification works, yet those small vectors are still meaningful and contain useful information, as opposed to artifacts such as noisy patterns. As indicated by the results, our algorithm achieves a clear and consistent improvement over random baselines. We hope this discussion can also inspire more research on vision-language dataset distillation.

(W2) Cross-architecture Generalization.

We report the cross-architecture generalization experiment results in the table below. Despite a performance drop when generalizing models across different architectures, as seen in existing literature on dataset distillation (Cazenavette et al, 2022, Deng & Russakovsky, 2022), the performance of 100 pairs of distilled data from NFNet+CLIP used for NF-ResNet50 and NF-RegNet remains significantly higher than the coreset selection baseline. We acknowledge the importance of cross-architecture generalization, but given that this is the first attempt at the vision-language dataset distillation, we expect that it is challenging for the distilled data to generalize across different architectures. When performing cross-architecture generalization for text, we could potentially operate on the nearest neighbor sentences rather than input embeddings. This approach could potentially offer a pathway for using distilled text data across different text base models.

We look forward to discussing these points further and appreciate the opportunity to clarify these aspects of our paper.