Open-Vocabulary Customization from CLIP via Data-Free Knowledge Distillation

Q1: As CLIP is already a very good domain-aware model, what is the motivation behind generating style tranferred images?

A1: It is precisely because CLIP is a very good domain-aware model that we can invert images from it with diverse styles. The diversity of training data is crucial for effective knowledge distillation. We analyze the generalization error of the inversed data through Theorem 4.1, providing a theoretical explanation for the benefits of diversification. We also prioritize generating realistic images. The style dictionary we devise aims to closely emulate real-world scenes.

As noted in [4], ensembling multiple prompts, including even random words, can improve CLIP’s performance. This has been interpreted as a form of noise augmentation, which enhances the robustness of model for a variety of domains.

Q2: Can pretrained diffusion models be used instead of VQGAN, as it can generate more diverse datasets very easily? What are the pros and cons of using a diffusion model?

A2: We also considered using diffusion models as generators. Unlike VQGAN, which maps $\boldsymbol{z}$ to the image $\hat{\boldsymbol{x}}$ in a single forward pass, diffusion models require a multi-step denoising process to generate images from noisy inputs or latent variables. Even advanced samplers like DDIM [5] or PNDM [6] typically require more than 10 steps. Performing model inversion involves optimizing input latent variables, which need to be differentiable. This means the full computation graph must be retained. However, most existing diffusion libraries (e.g., Hugging Face's Diffusers) use no_grad() during generation. To address this, we implemented a differentiable version of DDIM for experimentation, but found that even with just 5 steps, the computation graph exceeded 24GB of GPU memory. In summary, we choose VQGAN for its efficiency and effectiveness.

Q3: Why meta learning based knowledge distillation over traditional supervised learning? Any theorectical reason?

A3: There are two reasons for using meta knowledge distillation:

• "Logits-based knowledge distillation tends to overfit by mimicking the teacher model's outputs, neglecting the invariant information and thus reducing generalization." (lines 77-78)
• "When using the surrogate dataset for distilling knowledge from CLIP, synthetic images may not cover all semantic information of real images, resulting in a gap between the training and testing distributions. In other words, the covariate shift issue in DFKD is more significant than in knowledge distillation." (lines 284-287)

Generalization is crucial when working with synthetic datasets, and it benefits from the implicit gradient inner product of meta-learning [7]. Meta knowledge distillation minimizes the loss on current styles while ensuring that the optimization direction also yields improvements across other styles. By encouraging a gradient direction suitable for all styles, the student model learns shared representations across different styles. We have demonstrated its effectiveness experimentally (+1.54%) and analyzed the reasons for its effectiveness in Theorem 4.2.

TinyCLIP [1] and CLIP-KD [2] distill small CLIP models using relation, feature, gradient, and contrastive paradigms. Unlike our approach, they perform cross-modal distillation (students also have a text encoder) and rely on large-scale datasets such as LAION-400M, YFCC-15M, or CC3M+12M. LP-CLIP [3] trains a linear layer using pseudo-labels produced by CLIP, enhancing robustness through knowledge distillation on the training set. In contrast, we train the student network on synthetic data via meta-learning. We explore applying LP-CLIP’s mixture of data augmentations and consistency loss to our student model distillation process. As shown in the table below, consistency loss achieves comparable performance but slightly lags in generalization across different datasets.

[4] Waffling Around for Performance: Visual Classification with Random Words and Broad Concepts. ICCV 2023.
[5] Denoising Diffusion Implicit Models. ICLR 2021.
[6] Pseudo Numerical Methods for Diffusion Models on Manifolds. ICLR 2022.
[7] On First-Order Meta-Learning Algorithms. ArXiv 2018.

Thanks to the authors for rebuttal for addressing my concerns. The explanations are satisfactory. I would like to increase my score to 8.

Q: My concern mainly lies in technique novelty. Can you summarize your contribution again based on my concern?

A: Our work addresses real-world customization scenarios where users upload required texts or example images to meet practical tasks. We focus on improving the realism and consistency of synthetic images, expanding distributions based on few-shot images, and leveraging synthetic data properties to enhance the generalization of knowledge distillation. Below is a summary of our contributions based on your concern:

For dataset inversion：

• Synthetic images from [1] often lean toward artistic styles due to CLIP's training on internet data (see Figure 4). To address this, we redesign the style dictionary to better suit real-world applications. Experiments using the style dictionary from [2] reveal a saturation trend, where irrelevant styles introduce negative bias. In comparison, our strategy balances realism while maintaining diversity through optimized style prompts.

  	Caltech-101
  16 styles	57.99
  50 styles	60.78
  86 styles	60.49
  Our 16 styles	61.33

• We propose class consistency maintaining to prevent overly stylized deviations, which introduces and generates additional semantic information. The classification head, constructed with class semantics, acts as an anchor to regularize diversified data within CLIP’s embedding space. This method yields an average improvement of 0.6%.
• We innovatively propose image-based customization for inversion, constructing prototypes from example images to reduce intra-class variance. Instead of solely relying on few example images for knowledge distillation, we expand the distribution by leveraging the teacher model’s knowledge. This approach effectively reduces generalization error and enhances performance.

  	Image-based	Text-based	Improvement
  Caltech-101	84.78	61.33	23.45
  ImageNet	74.54	65.15	9.39
  Flower-102	74.72	18.07	56.65

Based on your suggestion, we revised the manuscript structure to better situate the contributions. Preliminary is now presented as Section 3. The first part rethinks BN-based model inversion, while the second part introduces the VQGAN-CLIP inversion paradigm. We also moved style dictionary diversification details into this section for better clarity.

For knowledge distillation:

Synthetic data inherently challenges generalization due to covariate shifts. To address this, we employ meta knowledge distillation on synthetic data with diverse styles. Our contribution lies in complementarily leveraging meta-learning and style diversification for data-free knowledge distillation. Meta-learning implicitly optimizes gradient inner product [4], a widely recognized technique for enhancing generalization [3,5,6]. By integrating this into our framework, we improve the effectiveness of knowledge distillation.

[4] On First-Order Meta-Learning Algorithms. ArXiv 2018.
[5] Gradient Matching for Domain Generalization. ICLR 2022.
[6] Implicit Gradient Alignment in Distributed and Federated Learning. AAAI 2022.

Apologies for missing the PDF upload deadline by just a few minutes—we had already finalized the revised manuscript, but unfortunately, it was slightly late.

In the DFKD paragraph of the related work section, we primarily discussed BN-based DFKD methods such as DeepInv, CMI, and Fast to introduce the subsequent Rethinking Model Inversion. Meanwhile, [2] and image-text matching were discussed in the introduction and preliminary sections.

Addressing your concern, we have revised the DFKD paragraph in the related work section as follows:

... They typically assume a classification model with a classifier that outputs logits. Target classes are represented as labels defined in the classifier's label space. (Original content introducing DeepInv, CMI, etc.) In contrast, [2] proposed leveraging image-text matching for DFKD in VLMs, introducing three prompt diversification methods to extract out-of-distribution capacity from VLMs. In comparison, our target classes can be represented not only as texts but also as example images. Additionally, we propose consistency strategies to prevent noise semantics introduced by styles and enhance realism.

This revised manuscript will be uploaded in the final version. Thank you again for your suggestions.

Q1: While Figure 1 provides a good overview of the framework, consider replacing the "frozen" and "not frozen" symbols with more intuitive icons, such as a lock and an unlocked lock.

A1: Thank you for your feedback. We have updated the symbols in Figure 1, replacing the "frozen" and "not frozen" symbols with more intuitive icons to enhance visual clarity.

Q2: Tables 2, 4, and 5 don’t have any units for the numbers or any text mentioning the metric used for those results. Please consider adding metrics and units.

A2: The numbers in Tables 2, 4, and 5 represent the classification accuracy (in %) of the student model. Thanks for your suggestions. We have clarified the metric in the experimental setup and added the units to the tables for consistency and clarity.

Q3: Why did you use VQGAN? Will the generated data have enough diversity? What other architectures did you consider?

A3: Directly optimizing in the image $\hat{\boldsymbol{x}}$ pixel space is possible, but images typically reside in high-dimensional spaces (e.g., 224 $\times$ 224 $\times$ 3), making the optimization problem challenging and computationally expensive. To achieve more efficient parameterization, we adopt the pre-trained VQGAN decoder and perform optimization in the latent variable space $\boldsymbol{z}$, which is low-dimensional. Compared with the pixel updating strategy that updates different pixels independently, the generator can provide stronger regularization on pixels since they are produced from shared weights.

We conduct the following experiments: directly inversing images without VQGAN, using VQGAN pre-trained on ImageNet (the VQGAN used in our paper), using VQGAN pre-trained on OpenImages. We also experimented with training VQGAN from scratch during the inversion process but encountered severe mode collapse.

As shown, using ImageNet VQGAN performs better on ImageNet than OpenImages VQGAN. The w/o VQGAN involves optimizing pixels directly for 400 iterations. While optimizing for thousands of iterations (as done in DeepInv) could improve results, this highlights the efficiency advantage of VQGAN.

We also considered using diffusion models as generators. Unlike VQGAN, which maps $\boldsymbol{z}$ to the image $\hat{\boldsymbol{x}}$ in a single forward pass, diffusion models require a multi-step denoising process to generate images from noisy inputs or latent variables. Even advanced samplers like DDIM [1] or PNDM [2] typically require more than 10 steps. Performing model inversion involves optimizing input latent variables, which need to be differentiable. This means the full computation graph must be retained. However, most existing diffusion libraries (e.g., Hugging Face's Diffusers) use no_grad() during generation. To address this, we implemented a differentiable version of DDIM for experimentation, but found that even with just 5 steps, the computation graph exceeded 24GB of GPU memory. In summary, we choose VQGAN for its efficiency and effectiveness.

[1] Denoising Diffusion Implicit Models. ICLR 2021.
[2] Pseudo Numerical Methods for Diffusion Models on Manifolds. ICLR 2022.

Q4: In Figure 6, the style differences are not very apparent—could the authors clarify how style diversification manifests visually?

A4: Style diversification is achieved by encouraging diverse prompts while retaining the core class semantics. These prompts lead to subtle but meaningful variations across images within the same class. For example, in the "Water Lily" and "Airplane" categories, differences in color tones and lighting effects can be observed. For "Pyramid," variations appear in surface textures, while in the "Valley" category, the viewing angles differ.

The style dictionary we devise aims to closely emulate real-world scenes. The selected words do not introduce additional semantics. We also experimented with a style dictionary containing styles sourced from the Internet and varied the number of styles.

Generally, increasing the number of styles enhances diversity. However, as shown in the table, we observe a saturation trend where irrelevant styles may introduce negative bias. This occurs because some styles inadvertently add class-irrelevant semantics, such as unrelated elements or complex backgrounds. To address this, we retain only words that describe photorealism and propose class consistency maintaining to prevent overly stylized deviation.

Q1: The relevance of BatchNorm (BN) statistics to the later-introduced contrastive learning. The presentation would benefit from clearer contextualization and integration with recent advancements in VLM customization.

A1: Previous studies rely on BN statistics but fail when applied to VLMs. In response, our Preliminary introduces an alternative inversion way: image-text matching, which does not require BN. Additionally, Appendix C explains why image-text matching outperforms BN-based methods. BN-based CMI [1] uses contrastive learning to ensure that images are distinguishable from previously synthesized images (line 125). In contrast, we use contrastive learning to enhance the diversity of text prompts, thereby increasing the diversity of synthesized images.

Inspired by your suggestions, we have revised the structure to better situate our contributions. The Preliminary section is now established as Section 3. The first part discusses our rethinking of existing BN-based model inversion, while the second part introduces the image-text matching advancements upon which our work is based. Following this contextualization in the Preliminary section, we transition into the Methodology section.

Q2: Are there specific aspects of CLIP’s architecture that are essential to this approach, or could it be adapted to other VLM architectures?

A2: Our method is applicable to VLMs with a vision encoder and text encoder structure. We also conducted inversion experiments on PyramidCLIP (which outperforms CLIP by 10%) and found that the issue of unusable BN layers is widespread; please refer to Appendix B for details. Based on your suggestion, we have also included the performance of our method on BLIP [2] and EVA [3] in the revised manuscript.

As shown in the table, EVA achieves the best performance, followed by CLIP and BLIP. The pre-trained weights we use for BLIP are the official blip-itm-base-coco. We observe that BLIP’s image-text matching capability is weaker than CLIP's. This is because BLIP's text encoder is image-grounded and trained on a binary classification task conditioned on images. Additionally, BLIP’s pre-training dataset contains 14M images, whereas CLIP is pre-trained on 400M text-image pairs. EVA achieves the highest zero-shot top-1 accuracy on ImageNet-1K, and demonstrates strong image-text matching capability. These results indicate that our method is applicable to other VLM architectures and that its performance improves with the capability of the underlying VLM.

Q3: While the proposed techniques are valuable, additional clarity around specific limitations—such as the potential for style dictionary diversification to introduce noise—could strengthen the paper.

A3: The potential for style diversification to introduce noise is clarified in both the methodology and experiments. To mitigate the potential noise introduced by style diversification, we propose class consistency maintaining (lines 270-283), which promotes consistency between synthetic images and their corresponding class text. It serves as an anchor to regularize the class semantics of the diversified data within CLIP’s embedding space. Table 1 demonstrates that the combination of style dictionary diversification and class consistency maintaining achieves the highest performance improvement. Furthermore, Table 4 provides an ablation study on consistency and diversity, illustrating the trade-off between the two.

[1] Contrastive Model Inversion for Data-Free Knowledge Distillation. IJCAI 2021.
[2] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. ICML 2022.
[3] EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. CVPR 2023.

Dear Reviewer fKMo,

Thank you for your thoughtful feedback on our paper! We have done our best to address your concerns and questions. We would greatly appreciate it if you could let us know whether our response has addressed your concerns.

We look forward to hearing from you.