Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance

We are grateful for your thorough review of our paper. Initially, we would like to offer the following responses, along with the list of planned action items.

===

Utilizing energy models to explain the fine-tuning of pre-trained models seems not to be essential. As per my understanding, the objective of the method in this paper as well as related methods ([1,2,3], etc.) is to reduce the difference in features extracted by the models before and after fine-tuning.

In the main text, we have covered the seminal work conducted by Kumar et al. (2022), where they posited that fine-tuning has the potential to modify the features acquired in pre-training. Furthermore, we explored practical methods for robust fine-tuning, including the contributions of Wortsman et al. (2022b) and Mao et al. (2022) you mentioned. The work by Mukhoti et al. (2023) also aligns with the concept of robust fine-tuning we discussed (note that Mukhoti et al. (2023) is contemporaneous, as outlined in the FAQ for Reviewers).

The idea you mentioned, which involves minimizing the discrepancy in features extracted by models before and after fine-tuning, represents a fundamental principle shared by various robust fine-tuning methodologies we discussed (e.g., weight-space regularization via EMA and L2SP, function-space regularization via KD and CAR-FT, and post-hoc weight-space regularization via WiSE and TPGM). Our work is significant in interpreting the success of these robust fine-tuning methodologies in the context of fine-tuning vision-language models based on a joint energy-based model (Section 4.2). Furthermore, we propose a novel approach called Lipsum-FT, building on this understanding (Section 5).

Ensuring that the vision model maintains the original inner products with features generated by the language model is our distinctive idea and undeniably possesses novelty. It is worth quoting the following by Michael Black: "If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea."

The authors claim that the text used is randomly generated, but it appears from the code in the supplementary material that tokens are sampled from the openai_imagenet_template. According to CAR-FT, using all templates as text input also yields good performance. What then is the significance of random token sampling in this scenario?

The authors claim that the text tokens are randomly generated. What are the specific rules for generation?

In the main text, we employ the zero-shot classification head, obtained from the open_ai_template, as the initialization for all fine-tuning methods. As a result, the open_ai_template present in the code is exclusively utilized for constructing the zero-shot classification head and is not involved in generating random texts for Lipsum-FT. The following lines of code, responsible for randomly generating text tokens using a vocabulary of size 49406, would clarify your concerns regarding the generation of random text:

            txts = [jnp.array(
                [49406,] + [0,] * config.token_length + [49407,]).at[
                    1:1+config.token_length
                ].set(jax.random.randint(
                    rngs[iii], (config.token_length,), minval=0, maxval=49406))
                for iii in range(config.token_k_ways)]

It is suggested that the authors provide a brief introduction to energy models in the related work section.

Thank you for your valuable suggestion! Including a brief introduction to energy-based models in the related work section is a great idea. It would provide readers with a better context for understanding our research. We appreciate your input and will supplement the Appendix A section (Action Item #1).

In Figure 1, it is not mentioned which points different learning rates in the left graph and different steps in the right graph correspond to.

While we mentioned in the main text that an increase in learning rates or additional training steps results in upward and leftward movement, it is advisable, as you recommended, to provide explicit specifications on plots to improve readability. We will accordingly revise Figures 1 and 6 (Action Item #2). Thank you for your constructive feedback!

===

We will share the upcoming action items once they are completed. If there are any remaining concerns, please let us know. Otherwise, we would like to ask you to raise your assessment accordingly.

• (Action Item #1) An additional paragraph regarding energy-based models in the related work section.
• (Action Item #2) Improving readability of Figures 1 and 6.

We appreciate the comprehensive review of our paper. At the outset, we would like to present the following responses, along with a list of intended action items.

===

I am not sure whether the novelty of this paper can be regarded as significant. It just introduces a regularization item to make the vision model keep the original inner products with features generated by the language model after fine-tuning.

We acknowledge your perspective but hold a distinct view on novelty. Our method extends established concepts in robust fine-tuning, where existing approaches share the fundamental principle of mitigating the fine-tuned model from deviating significantly from the pre-trained model.

As an illustration, Wortsman et al. (2022b) achieve robust fine-tuning by (A) empirically observing the fine-tuned model residing in the same pre-trained basin (as seen in Neyshabur et al., 2020) and (B) implementing a straightforward weight averaging strategy (referencing Izmailov et al., 2018; Wortsman et al., 2022a). Similarly, our work introduces a novel (A) exploration using feature distortion theory and a joint energy-based model (referencing Kumar et al., 2022; Grathwohl et al., 2019) and (B) feature regularization strategy utilizing random text in the context of fine-tuning vision-language pre-trained models.

Ensuring that the vision model maintains the original inner products with features generated by the language model is our distinctive idea and undeniably possesses novelty. It is worth quoting the following by Michael Black: "If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea."

The illustration organization of this paper is not clear enough. Therefore, even though the key idea is not so complicated, I find it difficult to understand the viewpoint quickly.

Thank you for your valuable feedback! We plan to draw a conceptual figure that showcases the overall idea of this work, as we think it could address this concern (Action Item #3). It will undoubtedly enhance the readability of the paper and aid readers in better understanding. We appreciate once again for the constructive feedback.

As the main contribution of this paper is to utilize the regularization term in (7) to minimize the energy gap in (6), please explain why the energy gap is chosen for improving the robustness of zero-shot model fine-tuning. Why is it defined to be the squared difference of two inner products as in (6)?

The foundational assumption guiding energy gap minimization is that current robust fine-tuning techniques alleviate the decline in "pre-trained connections" between vision and language models. The term "pre-trained connections" carries some ambiguity, and one of our primary contributions involves quantifying it based on the joint energy-based model. Consequently, the notion of "weakening pre-trained connections" can be characterized as a degradation in discriminative modeling between the two, evident in an increased energy gap. Accordingly, Lipsum-FT, our proposed method, employs a regularization term to minimize the energy gap, effectively accomplishing the objective of robust fine-tuning.

            txts = [jnp.array(
                [49406,] + [0,] * config.token_length + [49407,]).at[
                    1:1+config.token_length
                ].set(jax.random.randint(
                    rngs[iii], (config.token_length,), minval=0, maxval=49406))
                for iii in range(config.token_k_ways)]

[49406  4665 34356 35720 16917 40783 42215   971 12205 49407]
['<|startoftext|>', 'wy', 'prism', 'ayat', 'harold', 'ghead', 'seagulls', 'tt', 'rapid', '<|endoftext|>']
[49406 13212 20894 23105 19637 36399 36513 48145 20958 49407]
['<|startoftext|>', 'locks', 'sasha', 'tte', 'beingsalmankhan', 'aras', 'viswas', 'tote', 'aftermath', '<|endoftext|>']
[49406 38664  8670  2313  1162 43327  1281 48089  4326 49407]
['<|startoftext|>', 'amat', 'shield', 'league', 'gam', 'decree', 'air', 'protested', 'dog', '<|endoftext|>']

Thanks to the authors for their detailed responses, which have resorted some of my concerns. The newly added figure for illustration of the whole framework makes the idea obviously more clear and I suggest the authors add it to the main part of the paper for clarity. I also suggest the authors describe their idea in a clearer and simpler way since I agree with the viewpoint that a good idea is unnecessary to be intricate.

We are glad that our revisions have met with your satisfaction! Following your suggestion, we will place the illustration depicting the overall idea at the beginning of the text, which will undoubtedly aid readers' understanding. Thank you once again for your constructive feedback!

However, I need the authors give more explanation to my first question about their motivation of designing such form of regularization. It's notable that two image features given by the vision model may be quite different even if they give similar inner products with a text feature given by the language model, especially when the inner products are close to 0. In this case, the regularization cannot guarantee the two image features to be similar. Is it reasonable for the robust fine-tuning task? Why can the proposed method give better performance than simply regularizing the image features to be similar to their original values?

You pointed out that a distinction might exist between $\mathcal{F}\_{\theta\_{0}}(x)$ and $\mathcal{F}\_{\theta}(x)$ for the input $x$, even when employing the suggested Lipsum-FT regularization, as Lipsum-FT specifically regulates inner product values related to text features instead of "simply regularizing the image features to be similar to their original values." It is a valid point and, in fact, acts as the primary reason why Lipsum-FT can be effective for robust fine-tuning.

To be more specific, Lipsum-FT goes beyond simply reducing feature distortion in all directions (e.g., through the simple regularization like minimizing $||\mathcal{F}\_{\theta}(x) - \mathcal{F}\_{\theta\_{0}}(x)||\_2^2$). Instead, it explicitly addresses feature distortion that negatively impacts the pre-trained vision-language connection, quantified by inner product values related to text features. In other words, Lipsum-FT allows a degree of flexibility for the feature vector to undergo changes after fine-tuning (i.e., "a distinction might exist between $\mathcal{F}\_{\theta\_{0}}(x)$ and $\mathcal{F}\_{\theta}(x)$"), as long as it does not compromise the pre-trained vision-language connection (i.e., "Lipsum-FT specifically regulates inner product values related to text features"). Considering that a linear probing baseline, which completely freezes features, does not achieve satisfactory downstream performance, it is clear that we should allow a certain level of feature distortion for enhanced performance in downstream tasks, and it constitutes a fundamental challenge in robust fine-tuning—determining how to regulate the extent of distortion within an acceptable range. In response to this challenge, our Lipsum-FT regularization design maintains a suitable "certain level of feature distortion" by retaining the pre-trained vision-language connection, effectively addressing both reference and distribution shift performance in downstream tasks in the context of robust fine-tuning.

Here, we provide additional results for the simple regularization mentioned above, i.e., minimizing $||\mathcal{F}\_{\theta}(x) - \mathcal{F}\_{\theta_{0}}(x)||_2^2$, further underscoring the efficacy of our Lipsum-FT regularization method. Labeling this method FeatKD (Feature Knowledge Distillation, as it employs knowledge distillation on output features), the following table compares it for the ImageNet scenario:

We believe that our response comprehensively addresses your remaining concerns. If there are any outstanding issues, please inform us. The final version of the paper will incorporate the discussion on this matter. We appreciate your valuable insights that make our paper solid!

We appreciate your efforts in reviewing our paper. Firstly, we would like to provide the following responses, along with a list of planned action items.

===

The motivation is unclear. The authors mention a problem with cross-distribution shifts, but Lipsum-FT is a fine-tuning model.

We respectfully differ on the concerns about "unclear motivation." Our study is in line with recent progress in robust fine-tuning research (Radford et al., 2021; Pham et al., 2023; Wortsman et al., 2022b; Mao et al., 2022; Tian et al., 2023), addressing the examination and alleviation of performance trade-offs between reference and distribution shift data during the fine-tuning of large-scale pre-trained models.

What means of four distribution shifts DomainNet-{P, C, I, S}. Figure 1 fails to represent the distribution shifts.

DomainNet serves as a representative dataset for exploring techniques to manage out-of-distribution data. "DomainNet-{P,C,I,S}" denotes paintings, cliparts, infographics, and sketches, respectively. Detailed descriptions are available in Appendix B.4, and visual examples are provided in Figure 14.

Could you please elaborate further the meaning of the statement "Figure 1 fails to represent the distribution shifts"? Figure 1 depicts the occurrence of a performance trade-off between reference and distribution shift data following the fine-tuning of CLIP-ViT models, as discussed earlier in Radford et al. (2021).

There are too many contents that need to be referred to Appendix. Please reduce some important Appendix contents and put them into the text.

Due to the substantial number of experiments we conducted, there are numerous instances where readers should refer to the appendix, as you mentioned. While we endeavored to incorporate the main experimental results into the main text, typically placing complementary results in the appendix (e.g., presenting Figure 1 for DomainNet in the main text and Figure 6 for ImageNet in the appendix), there may be results in the appendix that merit inclusion in the main text. If you identify any such results, kindly inform us, and we will make the necessary adjustments to the paper.

It looks like capturing correlations between images and each language by calculating their inner product vectors in Equation 8. The meaning of Equation 8 wants to express is to match all the language information one by one or to match the current m-th information? If it matches M language information, how much does the algorithm complexity increase? What is the significance of matching with alonely m-th?

The process of computing the inner product between a batched feature vector with a shape of [batch_size, 768] and a guidance vector $[\mathcal{G}(\boldsymbol{t_1}),...,\mathcal{G}(\boldsymbol{t_M})]^\top$ with a shape of [M, 768] is comparable to the process of computing the inner product between the batched feature vector with a shape of [batch_size, 768] and a classification head $\mathbf{W}$ with a shape of [num_classes, 768] (768 here is the dimensionality of the extracted features from the ViT-B/16 model). Consequently, it does not require significant computational cost unless M is extremely large.

Lack of complexity analysis on the Lipsum-FT algorithm and its impact on the experiments.

Thank you for underscoring the significant concern regarding additional training costs, especially when utilizing large models. In the provided code, Lipsum-FT's implementation entails extracting the text model's output for random text at each iteration, necessitating the forward pass of the text model. However, the output of a fixed text model for entirely random text can be precomputed in practice (similar to CAR-FT). The distinction lies in that while CAR-FT employs 80 guidance vectors, Lipsum-FT, for example, may select 80 from a pool of 10,000 guidance vectors for each iteration. Consequently, we can implement Lipsum-FT using the same computational resources as KD and CAR-FT. We will provide additional analysis on training costs via wall-clock time, the most practical way to measure the cost (Action Item #5).

===

We will share the upcoming action items once they are completed. If there are any remaining concerns, please let us know. Otherwise, we would like to ask you to raise your assessment accordingly.

• (Action Item #5) An additional analysis on training costs.

Thanks for authors' efforts. I have another suggestion:

Authors only conduct discussions on the large-scale vision-language model based zero-shot learning (ZSL) methods, while the classical ZSL methods are neglected. I srongly encourage authors to take more classical ZSL methods [a-f] into discussions and comparisons, e.g., on ZSL benchmarks (CUB, SUN, AWA2).

[a] Improving Zero-Shot Generalization for CLIP with Synthesized Prompts. In ICCV, 2023.

[b] Evolving Semantic Prototype Improves Generative Zero-Shot Learning. In ICML, 2023.

[c] Closed-form Sample Probing for Learning Generative Models in Zero-shot Learning. In ICLR, 2022.

[d] HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning. In NeurIPS, 2021.

[e] Zero-shot learning by convex combination of semantic embeddings. In ICLR, 2013.

[f] Zero-shot Learning with Semantic Output Codes. In NeurIPS, 2009.

Thanks for authors' efforts. I have another suggestion:

Authors only conduct discussions on the large-scale vision-language model based zero-shot learning (ZSL) methods, while the classical ZSL methods are neglected. I srongly encourage authors to take more classical ZSL methods [a-f] into discussions and comparisons, e.g., on ZSL benchmarks (CUB, SUN, AWA2).

Following your suggestion, we carried out an additional experiment for generalized zero-shot learning using the CUB dataset. Since the conventional softmax classifier cannot handle unseen classes, we employed fixed zero-shot head weights derived from the pre-trained text model of CLIP. To be more specific, we first fine-tuned the vision model (with the fixed zero-shot head weights) on the training data for 150 seen classes. Subsequently, we assessed the performance of the fine-tuned vision model (with the fixed zero-shot head weights) on the test data that included 150 seen classes and 50 unseen classes.

The following table summarizes the results - S represents the accuracy for 150 seen classes, U represents the accuracy for 50 unseen classes, and H denotes the harmonic mean of these values - averaged from three trials. Additionally, for baseline comparisons, we incorporated CoOp and CoOp + SHIP results from Wang et al. (2023).

Our Lipsum-FT approach is more proficient at maintaining the original superior performance of the zero-shot model for unseen classes (i.e., 52.5 → 48.5, while FT exhibits 52.5 → 33.6). It provides clear evidence of the effectiveness of the Lipsum-FT regularization in preserving the pre-trained vision-language connection. Moreover, due to the nature of fine-tuning, where the parameters of the vision model are directly adjusted for the given training data, we observe a significant improvement in performance for seen classes compared to the prompt learning baselines (i.e., CoOp baselines). As a result, our Lipsum-FT achieved the best performance of H=61.1 across the table.

As the discussion period nears its conclusion, we appreciate your understanding of the time constraints that hinder thorough experiments within the given timeframe. Nonetheless, we are confident that the experimental results we provided for generalized zero-shot learning successfully address your remaining concerns. The final manuscript will include additional discussions and further experiments related to generalized zero-shot learning. Thank you for your time in reviewing our paper.

---

Wang et al., 2023, Improving Zero-Shot Generalization for CLIP with Synthesized Prompts.

Thank you for your constructive comments that make the paper solid. To begin, we would like to provide the following responses, along with a list of planned action items.

===

It would be beneficial if authors include visualizations of $\boldsymbol{v}\_{\boldsymbol{v}, \boldsymbol{\phi}}(x)$ and $\boldsymbol{v}\_{\boldsymbol{\theta}_0,\boldsymbol{\phi}}$ to provide an inituitive understanding of the distinctions between the original zero-shot model, the fine-tuned model with exisiting methods, and the fine-tuned model with the proposed emthod.

We appreciate your insightful suggestion regarding visualizations, offering valuable insights to readers. As these vectors have high dimensions, such as M=80, we project them into a two-dimensional space using t-SNE (Action Item #6). We have applied this approach to models fine-tuned through three methods -- FT, CAR-FT, and Lipsum-FT -- in the DomainNet scenario. The outcomes align with our expectations: FT shows the most significant distinction, followed by CAR-FT, and Lipsum-FT displays the least. We will include an additional section containing these results. Once again, thank you for your constructive comment!

Expanding the experiments to involve various domains as reference data for fine-tuning and using other domains for evaluation would enhance the comprehensiveness of the study. This approach can shed light on the adaptability and robustness of the proposed method in different real-world scenarios.

Although we are currently exploring two scenarios that include DomainNet and ImageNet, it is clear that conducting additional experiments with varied choices of reference data will enhance the paper. Therefore, we intend to showcase supplementary results using a different split as the reference data (Action Item #7). For instance, Trivedi et al. (2023) employed DomainNet-S as the reference data.

It remains unclear whether there exists a weight for incorporating the regularization term proposed in Eq. (7) into the loss function. In Appendix B.2, authors have discussed existing methods that involve weights related to the regularization term, such as $\lambda_{L2SP}$ in Eq. (10), $\lambda_{KD}$ in Eq. (12), and $\lambda_{CAR-FT}$ in Eq. (12). Authors have also detailed how to select these hyperparameters. However, there seems no mention of how the weight for the proposed method is determined.

We appreciate your valuable feedback on the hyperparameter setup. In response, we adjusted the weight values within the hyperparameter space, aligning with the approach used for CAR-FT. To elaborate, we swept over {0.1, 0.2, 0.5, 1.0, 2.0, 5.0} and set 1.0 for both DomainNet and ImageNet scenarios. All hyperparameters for each method were adjusted to achieve the best performance on the reference data, as out-of-distribution data should not be considered in the fine-tuning procedure. We will clarify this point in the revised version of the paper.

The precision of the standard deviation values in Table 2(b) should be improved. The values of NLL are presented accurately to two decimal places, whereas the standard deviation values are limited to only one decimal place, with many of them showing as 0.0. Ensuring consistent precision in reporting these values would enhance the clarity and reliability of the results.

Thanks for your thorough feedback on our experimental results! We inadvertently utilized the {:.1f} formatter for NLL, and we will revise tables to maintain consistent precision when reporting numerical results (Action Item #8).

There may be a typo in the gradient descent update rule presented in Eq. (9). It should be as follows: $\boldsymbol{\theta}\_t = \boldsymbol{\theta}\_{t-1} - \eta \nabla_{\boldsymbol{\theta}} L_{CE}(\boldsymbol{\theta})$. It's advisable for the authors to thoroughly review other equations to ensure they are accurately represented.

Thanks for fixing the typo! We will double-check other formulas for errors as you mentioned.

It would be interesting to know if the proposed method is applicable to various fine-tuning techniques, such as adapters, LoRA, and prompt tuning.

We believe our approach will effectively operate when features extracted from the vision model undergo changes for the same input. For instance, LoLA, where additional low-rank factors undoubtedly lead to changes in features extracted from the same input, will be compatible with the Lipsum-FT regularization. Conversely, for some prompt learning or adapter approaches, where features extracted from the same input remain unaltered, applying the Lipsum-FT regularization would not be straightforward.