Enhancing Domain Adaptation through Prompt Gradient Alignment

1. why the proposed gradient manipulation is better?:

Thank you for your insightful question. We would appreciate further clarification on your reference to the paper discussing the limitations of scalarization, particularly in under-parametrized setups, as it seems to suggest scalarization’s inherent inability to fully explore the solution space.

• Regarding your point of the efficiency of scalarization over multi-task learning (MTL) methods, we discuss this in lines 53-59 of the main text. In particular, we simply re-weight per-task gradients (similar to scalarization) instead of adopting multi task learning methods. In addition, the MTL methods in [33,92] are generic methods and are designed to optimize all objectives simultaneously. In contrast, we focus more on adapting our model to the target task rather than training a model that has universal performance across all domains or profiling the entire Pareto front. This further motivates our choice to use re-weighting source and target objectives' gradients.
• While studies [33, 92] demonstrate scalarization’s advantages over certain MTL methods like IMTL, MGDA, GradDrop, and PCGrad (ProGrad), they do not establish scalarization’s equivalence with all gradient manipulation techniques. In contrast, gradient alignment has been proven effective in learning invariant features [69], and penalizing gradient norms is widely recognized for enhancing model generalization [102, 18, 4, 90]. Empirically, we conduct some ablation studies on these two components in Table 6, the illustrative example, and gradient similarity experiments, to show that solely using scalarization is not enough.

2. We provide results of our methods using ViT-B/16, ViT-L/14 backbones on OfficeHome in Tables 3 and 5 in the attached PDF, following experimental setups in [R1, R2]. We can observe the superiority of our methods among all baselines while finetuning a small portion of the backbones.

3. Thank you for the suggested references. We will add them in the revision.

4. Caption for Table 1 and reference typos: Thank you for pointing these out, we will fix them in the revision. The experiment in Table 1 is on dataset ImageCLEF with I, P, C representing ImageNet ILSVRC 2012, Pascal VOC 2012, and Caltech-256, respectively, and ResNet50 backbone is adopted. The results in Table 1 in fact are from Table 3, we will add detailed discription for this experiment in the later revision.

We hope our response could help address your concerns, we are very open to further discussion.

[R1] Wang, Zhenbin, et al. "Landa: Language-guided multi-source domain adaptation.", arXiv 24

[R2] Singha, Mainak, et al. "Ad-clip: Adapting domains in prompt space using clip.", ICCV 23

whether the comparison with existing methods is fair

As we follow the experimental settings in [6, 15, 22, R4, R5, R6], we use their reported results which we believe have been verified to be fair when comparing old UDA methods and prompt-based methods. Specifically, they share the same vision backbone as prompt-based ones, e.g. ResNet, and are optimized to reach their full potential. Only one advantage of prompt-based methods is the use of additional textual information about class names when making predictions. However, this comparison setup is not only used in UDA, but also adopted in many works in other domains such as spurious debiasing [R7], continual learning [R1, R2]. Therefore, we believe that old UDA baselines are fairly compared. Additionally, we include comparisons with more recent CLIP-based baselines in Table 5 of the attached PDF. Here we finetune only a small portion of the model compared to other baselines, but still obtain competitive performance.

is there any reason why they fail?

Their underperformance could be attributed to the difficulty of using a single backbone to simultaneously learn domain-invariant and class discriminative features [22, 75]. This difficulty usually arises from the inherent entanglement between domain and semantic information [R3], which can be further amplified as more source domains are involved [6]. In contrast, we not only devote a set of shared prompts to learn domain-invariant features, but also domain-specific prompts to capture useful information for classification. By leveraging these prompts, we create a more meaningful representation, leading to more precise predictions for the target domain

would it be possible to evaluate the proposed method by updating the model

Yes, it is possible. Our proposed algorithm and theory development are generic and could be extended to finetuning the entire model. However, this would still entail a large computation and space complexity due to the size of the whole model, despite the avoidance of Hessian matrix calculation. Even so, adapting the full model on source/target domain can easily lead to overfitting and potentially inferior results in the presence of domain shift, even in the case of transformer models [1, 14, 34]. Therefore we opt for prompt learning which is a better way to adapt CLIP to downstream tasks (similar to other CLIP-based baselines) and is less likely to overfit.

Related works

Thanks for the suggestion, we will add them in the revision. Briefly, the second work, ProGrad, manipulates gradient of the fine-tuned loss to preserve general knowledge of the pretrained model. Although it shares the intuition of gradient alignment with our work, there is a significant difference: ProGrad attempts to remove conflicts between per-objective gradients at each time step, similar other gradient-based MTL methods such as [96, 39]. In contrast, we aim to stimulate their inherent consensus throughout training by encouraging the same training trajectory for both domains, hence, the model can find commonly good regions for them. The first work appears unrelated to ours as it approaches UDA from a architecture-view, i.e. introducing additional modules to generate content- and style-aware prompts, whereas we tackle it from the view of model optimization.

Difficulty of some formulas and the actual loss function

• Ideally, Eq.(9) and Eq.(16) would be the actual loss functions that we want to take derivative. The reason we approximate them to get Eq.(11) and Eq.(18) is to show that minimizing them can fulfill our purposes of (i) aligning prompt gradients and (ii) penalizing prompt gradient norms.
• Note that Eq.(16) indicates our full algorithm while Eq.(9) is only about gradient alignment.
• However, one difficulty in taking derivative of Eq.(9) or Eq.(16) is the computation of Hessian matrix, hence we devise a practical approximation as shown in Eq.(13) and Eq.(19), which allows us to only need to compute gradient of the original source/target loss, i.e. Eqs.(3,4), at the perturbed prompts.
• Please refer to the general response to see the actual loss function.

[R1] Wang, Yabin, et al. "S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning.", NeurIPS 22

[R2] Yu, Jiazuo, et al. "Boosting continual learning of vision-language models via mixture-of-experts adapters.", CVPR 24

[R3] Cai, Ruichu, et al. "Learning disentangled semantic representation for domain adaptation." IJCAI 19

[R4] Bai, Shuanghao, et al. "Prompt-based distribution alignment for unsupervised domain adaptation." AAAI 24

[R5] Li, Xinyao, et al. "Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation." CVPR 24

[R6] Du, Zhekai, et al. "Domain-agnostic mutual prompting for unsupervised domain adaptation." CVPR 24

[R7] Phan, Hoang, et al. "Controllable Prompt Tuning For Balancing Group Distributional Robustness", ICML 24

We would like to thank the reviewer for acknowledging our effort and we are encouraged that our responses have addressed most of your concerns. Regarding your last concern about the fairness of our experiments with traditional UDA methods, we have examined DANN and CDAN using a CLIP backbone to provide more insight into the limitations of domain-invariant feature learning in adapting pretrained CLIP to new domains. We will run more experiments on other datasets and incorporate obtained results to the revision. It is important to note that our baseline results, so far, are taken directly from prior work, as we adhere to the recent protocols for adapting CLIP from prior work.

Specifically, we applied DANN and CDAN on CLIP’s ResNet50 configured in three different ways: with a randomly initialized fully connected classifier (RN50-FC), with a frozen text encoder (RN50), and using the entire CLIP backbone (CLIP-RN50). The results, presented in the table below, demonstrate that appropriately adapting prior UDA methods to different parts of the CLIP model can yield better results compared to traditional methods on a ResNet backbone (e.g., DAN, CORAL, DANN). Eventhough, PGA and MPGA still exhibit superior performance, even with significantly fewer parameters finetuned. We hypothesize that relying solely on source classification loss and another objective for invariant feature learning can degrade CLIP’s rich semantic representation [R8, R9, R10, R11], which is crucial for predicting target domain data. To counteract this, utilizing target pseudo data (similar to self-training baseline in Table 1) or adopting a more carefully-designed optimization procedure that better leverages information from both source and target data—similar to our proposed method—could enhance performance.

We hope this response provides further insights into why traditional UDA methods may not perform as well as other prompt-based baselines. If the reviewer found any further unaddressed concerns, we are always happy to provide further clarifications and improve our work based on the constructive feedback from the reviewers.

[R8] Kumar, Ananya, et al. "Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution." ICLR 22.

[R9] Zheng, Zangwei, et al. "Preventing zero-shot transfer degradation in continual learning of vision-language models." ICCV 23.

[R10] Lai, Zhengfeng, et al. "Padclip: Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation." ICCV 23.

[R11] Ding, Yuxuan, et al. "Don't stop learning: Towards continual learning for the clip model." arXiv 22.

We sincerely thank the reviewer for the engaging discussion and greatly appreciate the valuable suggestions and feedback provided. We are honored by your appreciation of our paper and grateful for your support toward its acceptance. Thank you for your positive evaluation!

1. In Figure 1 of the attached PDF we provide the complexity for some comparative baselines. Accuracy curve (left): While DANN and CDAN obtain their best performance at approximately 77% after more than 1000s, PGA and MPGA achieve 84% within 100s. Besides, the first stage of pairwise source-target training of MPA takes 159s, followed by 35s for the second stage to actually train the final model. Number of Trainable Parameters (middle): PGA and MPGA, with fewer than 140k parameters, require significantly fewer parameters than MPA, DANN and CDAN, which have around 1M, 48.9M and 51.7M parameters, respectively. GPU Memory Usage (right)} PGA, MPGA, and MPA exhibit substantially lower memory footprints, around 1300MB compared to 7000MB of DANN and CDAN throughout training.

2. Thank you for the suggestion. Given the task-agnostic nature of our method, where PGA/MPGA utilizes only gradient information, and the successful application of CLIP-based methods in diverse tasks like image segmentation [R2] and object detection [R3], we believe our approach could be similarly effective in other tasks. For instance, applying our method to image segmentation, which requires predictions for each pixel, would necessitate training an additional decoder. Due to constraints in the rebuttal period, this extension is not straightforward, and is considered beyond the scope of our current experimental setup. We leave this for future work. Nonetheless, we believe that our method's superior results in image classification are sufficient to verify its benefits, as we adhere closely to the standard protocols of related UDA methods.

3. Figure 2 in the attached PDF shows that PGA is generally not sensitive to $\rho_{ga}$ and $\rho_{gn}$ within their acceptable range, i.e. 1e-2 to 10 for $\rho_{ga}$ and 1e-5 to 0.1 for $\rho_{gn}$. Specifically, (i) a too large value of $\rho_{gn}$ is less effective than smaller ones; (ii) ImageCLEF prefers larger values of $\rho_{ga}$ while OfficeHome prefers smaller ones, suggesting that source and target domains in the former dataset may be more similar than those in the latter, hence over-matching gradients in the latter dataset may be adverse.

4. It is true that when source and target domains are very different from CLIP's training data, prompt learning alone may not be sufficient to adapt well to the target domain. Yet, our method is still expected to perform better than prompt-based baselines. A failure case is discussed in our adaptation to the QuickDraw domain of DomainNet, which contains black and white sketches. Since this domain differs significantly from CLIP's training data, all prompt-based methods fail to achieve good accuracy and perform worse than some full-finetuning methods. This observation suggests integrating our methods with other invariant feature learning methods (lines 32-33 in our manuscript) or finetuning more parameters can better adapt to the target domain. Nonetheless, our PGA and MPGA still yield the highest results among prompt-based ones, indicating that our gradient alignment and norm penalization procedure is more helpful in closing domain gap than previous DAPL and MPA. Another example where prompt-based cannot surpass traditional UDA baselines is Rw→Cl from OfficeHome in Table 5.

   To conclude, for specialized domains not well-represented by the pretrained CLIP model, a more effective strategy might involve updating additional parameters, such as the layers close to the output in both vision and text encoders, beyond just the prompts. This could lead to better domain adaptation at the expense of computational complexity increase. We acknowledge this limitation in Appendix D and leave it for future work.

5. We test PGA on the setting of label shift following [R1], where the source or target domains are down-sampled with only 30% of data from the first-half of the classes are taken (indicated by s- prefix). Results presented in Table 4 in the attached PDF, table below and Table 5 in main text show the effectiveness of PGA on different levels of label shift. | Method | Ar→sCl | Ar→sPr | Ar→sRw | Cl→sAr | Cl→sPr | Cl→sRw | Pr→sAr | Pr→sCl | Pr→sRw | Rw→sAr | Rw→sCl | Rw→sPr | Avg | |-|-|-|-|-|-|-|-|-|-|-|-|-|-| | ResNet-50 | 41.5 | 65.8 | 73.6 | 52.2 | 59.5 | 63.6 | 51.5 | 36.4 | 71.3 | 65.2 | 42.8 | 75.4 | 58.2 | | DANN | 47.8 | 55.9 | 66.0 | 45.3 | 54.8 | 56.8 | 49.4 | 48.0 | 70.2 | 65.4 | 55.5 | 72.7 | 58.3 | | JAN | 45.8 | 69.7 | 74.9 | 53.9 | 63.2 | 65.0 | 56 | 42.5 | 74 | 65.9 | 47.4 | 78.8 | 61.4 | | CDAN | 51.1 | 69.7 | 74.6 | 56.9 | 60.4 | 64.6 | 57.2 | 45.5 | 75.6 | 68.5 | 52.7 | 79.8 | 63.0 | | IWDANN | 48.7 | 62.0 | 71.6 | 50.4 | 57.0 | 60.3 | 51.4 | 41.1 | 69.9 | 62.6 | 51.0 | 77.2 | 58.6 | | IWJAN | 44.0 | 71.9 | 75.1 | 55.2 | 65.0 | 67.7 | 57.1 | 42.4 | 74.9 | 66.1 | 46.1 | 78.5 | 62.0 | | IWCDAN | 52.3 | 72.2 | 76.3 | 56.9 | 67.3 | 67.7 | 57.2 | 44.8 | 77.8 | 67.3 | 53.3 | 80.6 | 64.6 | | PCT| 57.5 | 78.2 | 80.5 | 66.7 | 74.3 | 75.4 | 64.6 | 50.7 | 81.3 | 72.9 | 57.3 | 83.5 | 70.2 | | PGA (Ours) | 57.4 | 84.8 | 86.4 | 76.0 | 84.6 | 85.6 | 74.5 | 57.1 | 86.1 | 75.9 | 57.4 | 85.3 | 75.9 |

6. Scalability w.r.t dataset-size: Our experiments already included small (ImageCLEF), medium (OfficeHome), and large-scale (DomainNet, S2RDA) datasets. Furthermore, our proposed optimization procedure is not data-dependent, hence the increase in data size or dimension would not cause computational overhead for PGA and MPGA compared to other baselines.

[R1] Tanwisuth, Korawat, et al. "A prototype-oriented framework for unsupervised domain adaptation." NeurIPS 21

[R2] Wang, Zhaoqing, et al. "Cris: Clip-driven referring image segmentation." CVPR 22.

[R3] Wu, Xiaoshi, et al. "Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching." ICCV 23.

Hi,

Could you take a look at the authors rebuttal and finalize your rating?

Thanks, AC

1. Thanks for the suggestion, we will revise the abstract to reflect our technical contributions more clearly. Regarding the multiple objective optimization viewpoint, our MOO problem consists of per-domain objectives, motivated by the strong performance of the self-training baseline in Section 1. This empirical results show that optimizing target loss using pseudo data alone is already a strong baseline, which is not typically found in traditional UDA methods. They instead often combine source loss with auxiliary domain alignment objectives.

• As can be seen from the first term on the R.H.S of Theorem 4.1, minimizing these two terms will reduce the generalization error, thereby closing the gap between target population risk and source empirical risk. Furthermore, as we also minimize the latter through the cross entropy loss on labeled source data, the former will be reduced. This is the ultimate goal an UDA method aims to achieve, as low population target risk indicates a model having a good generalizability on target data.
• Intuitively, maximizing gradient alignment encourages the optimization trajectories to be the same for all domains, and minimizing gradient norm helps find flat regions for both shared and domain-specific prompts, which alleviates overfitting and generally leads to better performance on source/target data [4, 18, 60]. Altogether, performance on target domain can be improved.

• ${g_t^{src};g_t^{tgt}}$ should be derived from the same prompt, and in the theorem, to ease the proof, they refer to the whole prompt set $[P_{sh}, P_S, P_T]$ at time step $t$. Note that this does not contradict our gradient alignment strategy which is applied on the shared prompt only. Indeed, as we showed in Remark A.7, since $P_T$ is not involved when computing the source loss, we can write $g^{src}_t = [g^{sh,src}_t, g^S_t, 0]$. Similarly, $g^{tgt}_t = [g^{sh, tgt}_t, 0, g^T_t]$. Therefore, the inner product between $g^{src}_t$ and $g^{tgt}_t$ is equal to the inner product between $g^{sh,src}_t$ and $g^{sh,tgt}_T$.
• Extension of the theory to other spaces like image, feature, and output spaces: Please note that the second term in Theorem 4.1, which contains the KL-divergence between the two underlying source and target distributions $\mu, \mu'$, is the motivation for aligning these two distributions in image, feature or output space. Normally, this is done in feature space due to the rich representation captured by the feature extractor. For example, marginal feature alignment is adopted in [R1] and conditional feature alignment in [R2]. In the case of conditional alignment, one has to assign pseudo labels to target data using the learned source model [R3], which may lead to accumulation error. In our case, as we use zero-shot CLIP to predict pseudo labels, the error could be alleviated. Last but not least, as mentioned in the paper, incorporating our gradient matching and norm penalization into those methods fully reflects the two terms from the R.H.S of the bound, hence could further boost performance.

4. We provide an ablation varying the value of threshold $\tau$ in Table 2 of the attached PDF. Our methods are not sensitive to $\tau$, and the best result is obtained at a reasonable trade-off between the quantity and quality of pseudo data.
5. Thank you for your suggestion, we will add this paper to the related works and its results to the Office-Home experiment, where they obtain an average accuracy of 71.4 compared to 75.8 (ours).
6. We include the performance of CLIP-based models on two Synthetic-to-Real datasets S2RDA-49 and S2RDA-MS-39 in Table 1 of the attached PDF. PGA achieves the best performance among the baselines.

• When does the overfitting issue appear: Since we are optimizing multiple objectives across domains, naively minimizing them might lead to overfitting on some particularly easy-to-optimize or limited-data objectives, while forcibly minimizing them could harm other objectives.
• Why does penalizing gradient norms help: Briefly, in standard supervised training, minimizing gradnorm will lower the population risk, hence reduce overfitting. Similarly, in UDA, applying this on source and target data will lead to lower population risks on respective domains. Specifically, followed Theorem 1 in [55], we can upper bound these risks as: $[L_{\mu}(P), L_{\mu'}(P)] \leq \max_{||\epsilon_{sh}||\leq\rho_{gn}}[\max_{||\epsilon_S||\leq\rho_{gn}}L_S(P_{sh}+\epsilon_{sh},P_{S}+\epsilon_S)+f_S(||P||), \max_{||\epsilon_T||\leq\rho_{gn}}L_T(P_{sh}+\epsilon_{sh},P_{T}+\epsilon_T)+f_T(||P||)],$ where $f_S,f_T$ are strictly increasing functions. Furthermore, the worst-case source loss can be approximated as $\max_{||\epsilon_{sh}||\leq\rho_{gn}}\max_{||\epsilon_S||\leq\rho_{gn}}L_S(P_{sh}+\epsilon_{sh},P_{S}+\epsilon_S) \approx L_S(P)+\rho_{gn}(||g_{sh,S}||+||g_S||),$ and similarly for the worst-case target loss.

• "In Eq. (6)". Thanks for pointing this out. Yes, $L_T$ is the correct notation. Also in the definition of the gradient w.r.t target loss (in line 224), the correct notation should be $g_t^{tgt}$. We will fix these in the revision.
• In Eq. (20): In fact, it should be $L_S$, which is similar to $L_T$ in the approximation of Eq.(19). Informally, the PGA gradients corresponding to the source objective are the derivative of $L_S^{\text{PGA}}$ w.r.t $P$, which are then approximated by the derivative of the source loss at $L_S$ the perturbed prompts: $P_{sh}=P_{sh}-\rho_{ga}{b}+\rho_{gn}\frac{g_{sh,S}}{||g_{sh,S}||}, P_S =P_S+\rho_{gn}\frac{g_S}{||g_S||}$. (This approximation follows the derivation of Eq.(13)).
• Overall objective: please refer to the general response.

[R1] Nguyen, A. Tuan, et al. "KL guided domain adaptation.", ICLR 22

[R2] Nguyen, A. Tuan, et al. "Domain invariant representation learning with domain density transformations.", NeurIPS 21

[R3] Long, M., et al. "Conditional adversarial domain adaptation", NeurIPS 18