Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization

Thanks for the detailed feedback. Please find below our response to each comment.

Unconvincing statement: CoCoOp already observes overfitting with a method proposed to address it.

We humbly argue that we are the first to provide a ``comprehensive’’ study of overfitting across different finetuning baselines, datasets and OOD settings. Hence in the revised paper, we will tone down the point about being the first and instead focus on our contribution of being comprehensive. Algorithm-wise, instead of using input-conditional prompts in CoCoOp that implicitly tackles overfitting, we propose OGEN to directly reduce overfitting via unknown-aware optimization with OOD feature synthesis and adaptive model distillation. Our benefits are validated by the gains when applying OGEN on top of CoCoOp (Table 1 & 2).

Unclear model design: explain the rationale behind the attention method used to generate unknown image features.

Please refer to Section 3.2 (3rd paragraph) for the reasoning. At high level, to generate unknown image features from just the class name (using its text features), we found it's hard to directly learn the text-to-image feature mapping. This is backed by our ablations in Table 4 - see the poorly performing ``no extrapolation'' baseline. Then we choose to condition the feature generation process on some knowledge about known classes (i.e. the image/text features of kNN known classes), arriving at an extrapolation solution which is easier. The remainder of Section 3.2 introduces two extrapolation schemes, both naturally implemented by cross-attention. The attention mechanism is similar to measuring similarities between the unknown text (query) and kNN known text (key) features first, and then using such text similarities to extrapolate unknown image features from kNN known image features (value). We will clarify more on such rationale in the revised paper.

Missing related works.

We will add discussions and empirical comparisons with the suggested works in the revised paper. Comparing results for many of the new works can also be found in our “Response to common concern: comparing with the SOTA”.

Small performance gains in Table 1, and results of OGEN+SOTA methods.

Our observations from Table 1 are that 1) OGEN achieves notable improvements on new classes, 2) while its base class accuracy remains competitive. This is the desired behavior to us since we focus more on the new class generalization, although this may make the gains in the Harmonic mean of base and new accuracies seem smaller. Table 2 is another example where OGEN achieves convincing gains on new datasets without hurting the source ImageNet performance. For SOTA results, please refer to our “Response to common concern: comparing with the SOTA”.

Thanks for the positive feedback on our work and recognition of its novelty. To address concerns about SOTA results, we include more comparisons in our “Response to common concern: comparing with the SOTA”, where our method is found to consistently improve more recent methods, including the suggested Maple and KgCoOp.

The new results are quite impressive. I raise my score to 8 and advocate for publication.

In my original review I had commented that CoOp is much better than what the authors report if the authors make a good faith attempt at tuning that baseline instead of copying the number from previous publications. This point remains unaddressed, but I believe it should not factor into my review since I know so many papers in this area do not tune baselines.

Thanks for the feedback! Below is our answer to your main (and great!) question to hopefully address the novelty concern.

Compare with relay-based methods, e.g. sampling a subset from LAION-5B and using it for replay.

We have experimented with replay methods on the older but sufficiently large LAION-400M dataset. The goal is to compare our synthesized OOD features against the real replay data (sampled from LAION-400M) in terms of their impact on training regularization. Please refer to Appendix-Section D in the revised paper for experimental details, especially how we sample the replay data, including class filtering, sampling strategy and data size. Our high-level observations from Fig. 7 are that replay methods perform well with large data size, but suffer from low data efficiency as well as large memory cost (see detailed reasoning in Section D). By contrast, our feature synthesis approach avoids these issues by synthesizing hard OOD features on the fly. One future direction is to combine feature synthesis with replay methods to take advantage of their respective benefits of data efficiency and diversity. We will consider moving some of these discussions/results to the main paper to highlight the novelty and benefits of our feature synthesis approach.

Thank you for the constructive feedback and helpful suggestions for our manuscript. Regarding the comparison with state-of-the-art methods, please refer to our “Response to common concern: comparing with the SOTA”. For your other comments, below is our point-by-point response.

Move Appendix Fig. 5 to the main paper, poor paper writing, and missing objective function in Sec 3.3.

We appreciate the reviewer for highlighting these presentation issues. We will fix them as suggested, and try to identify and rewrite sections that are dense or difficult to follow as well. If you have other suggestions on areas that could be improved to help us understand where to focus, it would be really helpful.

Class extrapolation is unreasonable when there are large conceptual gaps between base and novel classes.

We agree the quality of class extrapolation highly depends on the semantic similarity between base and novel classes. We do observe faithfully extrapolated class examples when the semantic gap is small (Fig. 3 for example), but less faithful ones otherwise.

Among the many downstream datasets we have experimented on, most of them demonstrate shared regularities between classes (unlike CIFAR10), which gives rise to good extrapolation quality. Examples include datasets with a large number of conceptually close classes like FGVCAircraft and StanfordCars, as well as datasets with only a few but pattern-sharing classes like EuroSAT (satellite images) and DTD (texture images). The well-extrapolated OOD data on these datasets yield consistent, and sometimes large, gains on novel class performance, see Fig. 6(a).

On the other hand, there are datasets UCF101 and Caltech101 that have relatively large semantic gap between classes. Their data size is also small (13k and 9k respectively). Both of the two factors will negatively impact the class extrapolation quality. However, we still observe from Fig. 6(a) convincing gains on novel classes for UCF101 and Caltech101. The reason is twofold: 1) the abundance of extrapolated class examples alleviates overfitting on small datasets by scaling up the negative data, 2) extrapolated data from known classes, despite being noisy, can still serve as ``semi-hard'' negatives to regularize decision boundaries. One natural direction for improvement is to enrich the data used for extrapolation, e.g. by combining a living memory of diverse web data with the few training classes on downstream datasets.

Limited novelty and performance gains of ALMT.

In response, ALMT is not the main innovation of this paper, and we view ALMT as a surprisingly simple add-on to reduce overfitting. Note behind the adaptive window size in ALMT lies a well-grounded rationale, i.e. to reduce both overfitting and underfitting during model distillation (see texts before Eq. 7). Such MT adaptation does not require extensive hyperparameter tuning to gain improvements, which is non-trivial.

Performance-wise, Tables 3&6 verify that, ALMT in its simple form, can indeed improve both base (ID) and new (OOD) class performance. Notably, the improvement on base class is larger ($\approx$ 1%), which is particularly useful in our case since our OOD feature generator may slightly degrade the base class performance when improving OOD generalization. Overall, ALMT serves as an optional booster that helps to strike a good balance between the base and new class performance. We will add such discussions in the revised paper, with ALMT's role to be toned down accordingly.