Towards Realistic Unsupervised Fine-tuning with Vision-Language Models

We thank the reviewer for providing valuable comments and address your concerns in the following responses.

[Q1]. While this approach is innovative, it may be sensitive to the distribution of confidences within the unlabeled data. If the confidences are not well-calibrated or vary significantly across samples, it could affect the effectiveness of UEO.

[A1]. We agree on the significance of the instance weight $w(x)$. However, UEO has been empirically demonstrated to be not sensitive to confidences. To assess the calibration of instance weights, we utilize the AUC score as an indicator. For instance, when comparing the results on the DN (Avg.) dataset under open-partial shift with two different backbones (ResNet-50 in Table 4 and ViT-B/16 in Table 5), UEO consistently outperforms the CLIP baseline in terms of the accuracy (ACC) score. It is worth noting that improved calibration may not necessarily result in larger gains. Similar observations can be made when comparing different datasets under the same backbone.

[Q2]. One experiment that is missing would involve applying the method to VLMs that are not well-calibrated and observing its impact on the performance of the fine-tuned model.

[A2]. As obtaining a poorly calibrated VLM is challenging, we kindly refer the reviewer to [A5] @ Reviewer eDVD for a visual pre-trained source model (VM). Carefully comparing the gains of UEO over the initized model in two following tables, we find that the AUC scores vary a lot across different models, and UEO always increases in terms of the ACC score.

[Q3]. When compared to InfoMax, there does not appear to be a significant improvement in accuracy on certain datasets. For instance, on the DomainNet dataset, there is no noticeable difference in performance in accuracy.

[A3]. We respectfully disagree. As stated in [A2] @ Reviewer pTa8, UEO consistently outperforms InfoMax when considering both accuracy (ACC) and area under the curve (AUC) together. While the difference between UEO and InfoMax on the DomainNet dataset may appear small, UEO consistently exhibits superior AUC scores under different category shifts and backbones.

[Q4]. Can the universal entropy optimization loss be potentially incorporated during the contrastive pre-training phase as well?

[A4]. That sounds like a reasonable and interesting direction for future work. When we have paired text-image pairs along with some unpaired images during the pre-training phase, we can take the texts from the pairs as pre-defined class names and incorporate a simialr loss on these unpaired images with the contrastive pre-training loss over image-text pairs.

[Q5]. Apart from being hyper-parameter free, could the author elaborate more on the benefit of using the proposed universal entropy optimization instead of the entropy separation loss in DANCE?

[A5]. In [A4] @ Reviewer eDVD, we show the disadvatnages of thresholding-based methods. Thresholding-based methods, including the entropy separation loss in DANCE, indeed rely heavily on the chosen threshold, and their performance can be sensitive to the absence of OOD samples, especially in scenarios like closed-set and partial-set category shifts where OOD samples are not present. Mistakenly classifying hard ID samples as OOD samples can introduce risks and impact the overall effectiveness of these methods.

We thank the reviewer for providing valuable comments and address your concerns in the following responses.

[Q1]. I believe more general classification datasets (e.g., ImageNet and SUN397 as in CoOp) should be employed to define the task setting and verify the efficacy of the proposed method.

[A1]. A great advice. To validate the effectiveness of the proposed method (UEO), we additionally utilize three widely recognized classification datasets (i.e., ImageNet, SUN397, and Food101) and present the results under four category shifts in the tables below.

[(C): closed-set, (P): partial-set, (O): open-set, (OP): open-partial]

On the ImageNet dataset, UEO and UPL are top two methods when considering both ACC and AUC simultaneously. Similarly, UEO and InfoMax are top two methods on the SUN397 dataset. Notably, UEO achieves the best performance on the FOOD101 dataset. In summary, UEO consistently attains the best or competitive performance across different datasets, while other methods exhibit more variability across the datasets.

[Q2]. The introduced method bears a significant resemblance to existing mutual information maximization losses. Additionally, the performance gains seem rather marginal when compared with its peers.

[A2]. We acknowledge that a direct comparison of the objective in Equation (4) with InfoMax [Liang et al., ICML 2020] reveals the sole difference lies in the instance weight $w(x)$. However, it is crucial to note that the motivation behind this choice is entirely different. We explain the relaxation from Eq. (3) to Eq. (4) in [A3] @ Reviewer eDVD. Concerning the performance gains, we collect the average results of InfoMax and UEO from Tables 1-4 in the table below. It appears that UEO outperforms InfoMax in a majority of cases, with only a few cases where it is inferior to InfoMax.

[(C): closed-set, (P): partial-set, (O): open-set, (OP): open-partial]

[Q3]. I'm uncertain if the competing methods in Tab. 1 also implement the same parameter-efficient tuning strategy. Furthermore, the prompt and affine parameters of the parameter-efficient tuning should be dissected and examined individually.

[A3]. We feel sorry for any confusion. As written in the last paragrpah in Sec 4.1, all the methods are implmented with the same parameter-efficient tuning techique. In particular, we reduce the learning rate for EntMin due to its rapidly decreasing results.

To verify the effectiveness of the proposed parameter-efficient tuning strategy, we present results with only text prompt tuning in the Appendix (Tables 14-17). Additionally, we investigate the influence of different combinations of parameters during the fine-tuning process in Fig. 3. In these figures, the first two columns denote optimizing the text prompt and visual affine parameters, respectively.

[Q4]. Missing references [1,2,3] about domain adaptation?

[A4]. Thanks for your kind reminder. [r1] explores the model adaptation problem in a black-box manner, [r2] considers domain adaptation under open-partial category shift, and [r3] addresses domain adaptation under partial-set category shift. These papers represent various domain adaptation problem setups, and we will include them in the related works section. Given that these methods are tailored to specific category shifts, which significantly differ from the problem setup presented here, direct comparisons with UEO in the experiments prove challenging.

• [r1] Zhang, Haojian, et al. "Unsupervised Domain Adaptation of Black-Box Source Models." In Proc. BMVC 2021.
• [r2] You, Kaichao, et al. "Universal domain adaptation." Proc. CVPR. 2019.
• [r3] Cao, Zhangjie, et al. "Partial adversarial domain adaptation." Proc. ECCV. 2018.

We appreciate the reviewer for providing valuable comments, and we address your concerns in the following responses.

[Q1]. What is the special role of the "vision-language model" in the paper or the investigated problem? It seems to me the approach and problem-setting are applicable to models beyond CLIP?

[A1]. Firstly, our approach aligns with established practices in prior studies, including references [r1, r2], where we explore the application of vision-language models with a specific focus on the CLIP model. Secondly, it is true that our method and the associated problem setup remain reasonable even when utilizing other pre-trained networks, such as traditional model adaptation. We present the corresponding results in [A5] @ Reviewer eDVD. However, the utilization of vision-language models like CLIP proves more apt for the proposed Unsupervised Universal Fine-Tuning problem. This is attributed to the advantage of not requiring labeled data collection efforts for training, thanks to the pre-defined list of class names.

• [r1]. Zhou, Kaiyang, et al. "Learning to prompt for vision-language models." International Journal of Computer Vision 130.9 (2022): 2337-2348.
• [r2]. Du, Yu, et al. "Learning to prompt for open-vocabulary object detection with vision-language model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[Q2]. I think the authors need to better clarify how their experiment setting differs from previous works. I am wondering if this is a new setting, or some previous setting adapted to CLIP, or some other cases?

[A2]. We feel sorry for any confusion. We would like to clarify that the proposed unsupervised universal fine-tuning problem is not derived from other related topics but constitutes a new and intriguing task. Since the proposed task is closely related to several tasks in the domain adaptation area (i.e., model adaptation and universal domain adaptation), we employ several popular benchmarks (e.g., Office, OfficeHome, and DomainNet) in the domain adaptation area. We provide additional results on more general benchmarks in [A1] @ Reviewer pTa8.

To the best of our knowledge, our task represents the first exploration into both ID generalization and OOD detection for unsupervised fine-tuning under unknown category shifts. Unlike most prior domain adaptation methods that assume knowledge of the category shift in advance and predominantly emphasize the ability of ID generalization, our task is more realistic. In the context of universal domain adaptation tasks, where the target data for adaptation and the test data share the same label space, evaluating the ability of OOD detection becomes challenging (as depcited in Evaluation of Sec. 3.2). Instead, our task introduces a distinct fine-tuning and evaluation protocol, along with different evaluation metrics.

[Q3]. Additionally, the numbers in Table 1-4 are quite close in some datasets (though I admire and appreciate the exhaustive evaluation from the authors), so it is also better to make more clarifications. (four questions ...)

[A3]. For each question, we offer a point-to-point response below.

• For the proposed parameter-tuning stragey, we show the results of only prompt tuning in Fig. 3 and Tables 14 - 17 in the Appendix. Regarding the objectives, EntMin and InfoMax could be considered as two baselines for UEO. Further comparisons with two-step thresholding variants are elaborated in [A4] @ Reviewer eDVD.
• Regarding the variance of the numbers, we conducted all the experiments using different seeds and observed a relatively small variance, approximately 0.1 for both ACC and AUC. Therefore, we omitted the variance in these tables
• Generally, DomainNet could be considered as the largest and hardest dataset in this paper. For instance, the Real domain in the DomainNet dataset comprises approximately 120k images across 345 classes. Notably, the ACC and AUC scores of the original CLIP model are consistently the lowest among all the datasets.
• In some datasets, UEO might not exhibit a clear superiority over the baselines, primarily for two reasons. Firstly, unsupervised fine-tuning is inherently challenging, and the majority of fine-tuning methods do not significantly outperform the original CLIP model. Secondly, enhancing both ACC and AUC scores concurrently is a difficult task. In most situations, a trade-off exists between these two scores. Notably, UEO consistently improves the ACC score without compromising the AUC score, outperforming other methods across the majority of datasets and category shifts.

Dear reviewer,

We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights before the end of the Author/Reviewer phase. Please let us know if any aspect of our work remains unclear or if you have additional feedback.

Thank you.

[Q5]. The proposed method appears to be a general OOD method that can be also applied to traditional classification networks. I wonder why this method is applied to only CLIP method instead of extending it to other pre-trained backbones.

[A5]. Indeed, our objective in Eq. (4) is adaptable to traditional classification networks as well. The reason why we focus on CLIP lies in its flexibility, as a pre-trained model can be easily acquired based on class names without the need for labeled data, rendering it a versatile framework.

Regarding the traditional classification networks, we employ the ResNet-50 backbone with the [backbone - bottleneck layer - classifier layer] architecture to train the source model. During unsupervised fine-tuning in the target domain, we only update the parameters of the bottleneck layer (parameter-efficient), the results on two different tasks are shown below. On both tasks, UEO significantly outperforms other methods in terms of both ACC and AUC metrics.

We thank the reviewer for providing valuable comments and address your concerns in the following responses.

[Q1]. The resemblance to previous methods [1,2,3] would compromise the novelty of this study.

[A1]. After carefully reviewing these papers [1,2,3], we observe that each of them utilizes a similar strategy, aiming to minimize the entropy of ID samples and maximize the entropy of OOD samples to enhance OOD detection capability. While our method does share a similar high-level idea, it diverges from these approaches in two key aspects.

The first difference lies in the problem setup, as our fine-tuning process is executed solely with unlabeled data, without explicit ID and OOD samples. In certain category shifts, such as closed-set and partial-set scenarios, OOD samples may not even exist in the unlabeled data, a condition that was not feasible for previous methods.

The second difference lies in the ultimate objective. If we know $w(x)\to 1$ for ID samples and $w(x)\to 0$ for OOD samples for unlabeled data in advance, Eq. (3) exhibits similarities to prior OOD detection methods [1,2,3]. However, our proposed method optimizes the objective in Eq. (4), representing a more relaxed yet robust alternative to Eq. (3).

[Q2]. However, a similar principle of maximizing Mutual Information ID samples and penalizing the mutual information of OOD samples has been also proposed in [4].

[A2]. Apologies for any confusion. Our method is different from [4]. Firstly, the method in [4] adopts the meta-learning framework that splits ID and OOD samples in the meta-training step, retaining the ground truth knowledge of ID and OOD during training. Secondly, the Mutual Information Maximization training objective in [4] is defined as the binary cross-entropy loss between ID and OOD prediction, which is distinct from our entropic loss. To be honest, minimizing the scores for ID samples and maximizing the scores for OOD samples a straightforward solution given the presence of ID and OOD samples in the training process. However, in the context of unsupervised fine-tuning, a key challenge arises in distinguishing ID from OOD samples. To circumvent the need for sensitive thresholding operations, we propose the relaxed optimization objective in Eq. (4).

Beyond the unified objective in Eq. (4), our paper introduces a novel unsupervised fine-tuning problem setup that considers both ID generalization and OOD detection. This is especially crucial in handling various category shifts between the training data and the pre-defined label set, making our fine-tuning approach a notable contribution to the field. particularly in the face of various category shifts between the training data and the pre-defined label set. We consider such a fine-tuning problem setup to be a significant contribution to the field. Additionally, we provide a new parameter-efficient fine-tuning strategy for CLIP that optimizes not only the text prompts but also the normalization layers in the visual branch.

[Q3]. How do Eq. (3) and Eq. (4) theoretically satisfy the principle "minimize the entropy of ID instances and maximize the entropy of OOD samples"?

[A3]. As explained above, when $w(x)\to 1$ for unlabeled ID samples and $w(x)\to 0$ for unlabeled OOD samples, Eq. (3) becomes the optimal objective that minimizes the entropy of ID instances and maximization the entropy of OOD samples. In the context of the unsupervised fine-tuning problem, where binary labels ("ID" or "OOD") are not available, we suggest employing the maximum softmax probability as an approximation for the ideal weight, as outlined in Eq. (3).

As the estimated weight $w(x)$ may not always be accurate — for example, no OOD samples exist in the unlabeled data under the closed-set and partial-set category shifts - maximizing the entropy can be harmful. Intuitively, we could maximize the entropy of mean prediction over OOD samples instead of the average entropy of individual OOD samples, leveraging the inequality $H(\mathbb{E}(p(x))) \geq \mathbb{E}(H(p(x)))$. Honestly, optimizing Eq. (3) and Eq. (4) does not theoretically guarantee the principle in the challenging unsupervised fine-tuning problem, but we believe the proposed objective in Eq. (4) establishes a simple yet strong baseline, as evidenced in extensive experiments.

[Q4]. The explanation is also missing of how Eq. (3) & (4) would be more suitable than a simple stepwise function, e.g. $L_{ID}=H(p(x))$ and $L_{OOD}=-H(p(x))$, and the determination of OOD samples follows the common practice as introduced in Sec. 3.1.

[A4]. Thanks for the advice. As written in the common practice in Sec. 3.1, $\lambda$ is chosen so that a high fraction of ID data (e.g., 90%) is above the threshold. Given the absence of explicit ID data in the unsupervised fine-tuning problem, we exclude it as a baseline method, denoted as EO-ID, in our experiments.

Following the interesting view from the reviewer, we assume $w\sim \mathcal{U}(\frac{1}{C},1)$, then obtain $\mathbb{E}(w)=\frac{1-\frac{1}{C^2}}{2}\times\frac{1}{1-\frac{1}{C}}$ and $\mathbb{E}(\frac{1}{w})=log(C)\times\frac{1}{1-\frac{1}{C}}$. To discover the sample whose entropy is minimized in Eq. (3), we have $\hat{w}(x) - \hat{\Phi}(w(x))>0 \Rightarrow \frac{w}{\mathbb{E}(w)} > \frac{1/w}{\mathbb{E}(\frac{1}{w})} \Rightarrow w^2\times \mathbb{E}(\frac{1}{w})> \mathbb{E}(w) \Rightarrow w > \sqrt{\frac{1-\frac{1}{C^2}}{2log(C)}}$. By the way, the threshold $\lambda=\sqrt{\frac{1-\frac{1}{C^2}}{2ln(C)}}$ sounds reasonable since it decreases when the number of classes ($C$) increases. We refer to the hard thresholding variant utilizing this implicit threshold as EO-H.

Here we conduct a simple comparison on the first domain (Ar) of the OfficeHome dataset and show the results across different category shifts below (EO denotes the objective in Eq. (3)).

[(C): closed-set, (P): partial-set, (O): open-set, (OP): open-partial]

As can be seen from the table above, UEO consistently outperforms all three thresholding methods in terms of ACC while being inferior to them in terms of AUC under open-set and open-partial shifts. Notably, the thresholding technique proves harmful for closed-set and partial-set shifts where no OOD samples exist in the unlabeled data. In summary, UEO demonstrates superior suitability compared to these thresholding variants.

Dear reviewer,

We genuinely appreciate the time and effort you've invested in reviewing our paper. We have carefully provided relevant responses and results to your concerns. We are eager to further discuss with you and gain your insights before the end of the Author/Reviewer phase. Please let us know if any aspect of our work remains unclear or if you have additional feedback.

Thank you.

We express our sincere gratitude to the reviewer for providing us with the opportunity to address the remaining concerns during the author/reviewer phase.

To validate the effectiveness of both the weighting strategy in Eq. (3) and the relaxation (derivation) strategy in Eq. (4), we extend our comparisons to three additional domains within the OfficeHome dataset. We present the averaged results across various category shifts [(C): closed-set, (P): partial-set, (O): open-set, (OP): open-partial] and domains [OH (Ar)/ OH (Cl)/ OH (Pr)/ OH (Re)] for different variants in the table below. Note that, UEO-H/ID replaces $\mathbb{E}(H(p(x)))$ in EO-H/ID with $H(\mathbb{E}(p(x)))$.

We can derive three main conclusions:

• Although the weighting strategy does not enable EO to surpass EO-H in Eq.(3), comparing UEO with UEO-H demonstrates the effectiveness of the weighting strategy in the final objective.
• The relaxation (derivation) strategy significantly improves all three variants (EO-ID, EO-H, and EO), with a notable impact on the ACC metric.
• UEO achieves the best performance while considering both ACC and AUC metrics.

 

We attach the detailed results under each category shift and each domain as follows.

[(C): closed-set, (P): partial-set, (O): open-set, (OP): open-partial]

 

 

 

 

Dear reviewer,

Concerning the only issue raised about the convincing derivation from Eq. (3) to Eq. (4), we have presented comprehensive comparisons between UEO and various variants in the rebuttal phase. We are eager to understand if these results have successfully addressed your reservations regarding the effectiveness of both the weighting strategy and the relaxation strategy in Eq. (4).

As for the other concerns that were not reiterated in your recent response, we would appreciate confirmation on whether they have been satisfactorily addressed in our previous response. We would be very grateful if you could provide your feedback before the end of the rebuttal phase.

The authors