Domain Prompt Matters a Lot in Multi-Source Few-Shot Domain Adaptation

Many significant problems are found in the current format of the paper that prevents the understanding of the concept. The problem includes but is not limited to confusing writing, inconsistency of notations, expressions of novelty, experimental presentation etc. It is highly recommended that the authors to re-write the paper, re-organize the content and better polish the text for the reader to better understand.

The major problems:

• Novelty is limited:

  • The proposed method as in Sec. 3.2 is very similar to DAPL but extended to multi-source scenarios. The only difference is to introduce an additional [DOM]. Note, that DAPL has not been peer-reviewed.
  • The motivation for domain-aware mixup is confusing. I cannot be convinced and do not understand in the current writing, how it can enforce to learn domain-specific knowledge. The corresponding literature regarding mixup in the feature space is also not referenced and discussed (e.g. [1]).
  • The description for deriving the domain-aware mixup is confusing. I assume the authors are trying to develop a method so that the learned prompt shares the knowledge between source and target domains (depending on Eq. 9)?

• Writing:

  • In the first sentence of Abstract: “large vision-language models … strong performance in MFDA”. There is no such reference applying large VL models in MFDA. In fact, MFDA is a rarely studied problem.
  • The description of the problem setting (MFDA) should be clearly explained at the beginning (abstract or introduction) so that the reader can refer better to the limitations of prior works.
  • Paragraphs 1 & 2 in the introduction: the connection is missing, and ‘prompt learning’ suddenly jumps in, making the concept broken.
  • Fig. 1 is not referred to in the paper.
  • Related work: after describing the related prior works of each field, it's suggested to write a couple of sentences to distinguish between them to show the novelty of the proposed method.
  • The description of the MFDA setting is very confusing in the first paragraph of the Method Section: “single target domain with sparse labels”, “…target distribution p_T(x, y) with label observation…” is mentioned, but the notation for target domain \tau is unlabeled. In the original MFDA paper (Yue et al., 2021a), the target data is unlabeled. What about the unlabeled data in source domains? Are they used during training (as in (Yue et al., 2021a))? It is very confusing that the problem setting description defers significantly as in (Yue et al., 2021a).
  • There is significant text overlapping with DAPL in the preliminary sections of both papers (only with some rewording..). It should be strictly prohibited.
  • What is [DOM] in Eq. 4? I assume it is a domain ID? And I assume [DOM] is the non-learnable component near the description of Eq. 4?
  • Notation: what is subscript d in Eq. 4 and superscript d in Eq. 5? They are not explained in the text. I assume they are the domain IDs?
  • What does it mean by ‘d*k categories’ as in the sentence after Eq. 5?
  • Eq. 6 is very confusing. For the outer summation on d \in {s, u}, what is the purpose of computing the similarity between the target domain prompt and source image features? How does the learning on unlabeled target data is realized?
  • What is inter-source domain mixup? In the current format of writing, I don’t understand why maintaining it will harm the representation learning on the target domain. The motivation is weak.
  • In the second paragraph on page 6, the notation of target domain data y_t is different from Section 3.
  • In Fig. 3, letters v and f are used to represent the features of “painting” and “real”. But v is used to represent text prompts as in Eq. 3
  • The feature mix-up formulation in Fig. 3 is different than Eq. 8. One uses \gamma and another one uses \lambda? and the weighting is different?
  • It is really confusing that the letter “t” is used to refer to text and target domain.
  • What are D^s and D^u in Eq. 10? They are never defined. I assume they are source and target domains, which is inconsistent with what is described in the problem setting. The problem setting is borrowed from (Yue et al., 2021a). But Eq. 10 is copied from DAPL paper. Please keep everything consistent throughout the paper. Also, Eq. 9 requires source data as well, why only D^u is passed to L_u as in Eq. 10?
  • The notations for loss functions in Eq. 7, 9, and 10 should be consistent.
  • Table 5 in the last sentence of Page 8 should be Figure 5.
  • The experimental setting/comparison is very confusing. What is “single best”, which can be both setting and method as in Table 1&2? What is source combined? Which rows in Tables 1&2 refer to the MFDA? How come the “Large model” in Table 1&2 can be the setting, it should be the model architecture.
  • For Figure 6&7, they are hard to see the differences. It is suggested to use a table to report the numbers.

[1] Adversarial Domain Adaptation with Domain Mixup. AAAI 2020.

• The paper could benefit from more ablation studies analyzing the effects of different prompts.

• The difference between the proposed method and existing works isn't clearly demarcated.

• Implementation details and settings for the backbone model are missing.

• Prepending and learning domain-specific or instance-specific prompt tokens to handle distribution shifts is not a novel idea. For example, to improve generalization to OOD data, the prompt vectors are conditioned on image inputs at test time [1, 2, 3]. Additionally, the idea of dividing prompts into domain-specific and domain-agnostic parts is introduced in [4]. Specifically, [4] heuristically partition the prompts into domain-specific and domain-shareable components. During adaptation, a manually crafted regularization term is employed to preserve the domain-shareable part while allowing the domain-specific component updates.

[1]Conditional prompt learning for vision-language models. CVPR 2022

[2]Domain Prompt Learning for Efficiently Adapting CLIP to Unseen Domains. 2022

[3] Test-time prompt tuning for zero-shot generalization in vision-language models. NeurIPS 2022

[4]Decorate the newcomers: Visual domain prompt for continual test time adaptation. AAAI 2023

• The paper falls short in providing a thorough review of related work. The previous works on domain adaptation with prompt learning between 2022 and 2023 are missed. Several works are mentioned above, and thus I suggest the author undertake a more exhaustive literature review and highlight the differences between the proposed method and those previous works.

• The presentation of the problem setting and motivation requires further clarification. While the paper puts forth a solution for few-shot domain adaptation, it doesn't clearly delineate the inherent challenges of the few-shot setting. Readers might wonder: How do limited labels influence the model's training? Specifically, does this scarcity impact the modeling of multi-source domains or the transfer of knowledge from the source to the target domain?

• The claims in the paper lack evidence. The author mentioned the extraction of domain information becomes more complex in multi-domain settings and using learnable prompts increases the risk of overfitting on training samples. However, those claims are neither supported with references nor empirical results.

• The writing of the introduction may pose challenges for readers who are new to the subject because the author mentions many terms but without adequate explanation. For example, what are context prompts? What are the differences between the domain and the semantics? What are context variables and domain variables? What is cross-domain mixed feature information?

The paper is challenging to follow, and many of the concepts and motivations are either incorrect or unclear.

Several claims that are unclear and unsupported:

1. “Using learnable prompts increases the risk of overfitting on training samples, which reduces the ability of domain prompt models to extract common semantic features. “ . This is not justified and likely wrong. If training one prompt for a domain do not have the issue, then training multiple prompts respectively for multiple domain should not have the issue as well.

2. “Large-scale pre-trained vision-language models lack domain awareness during training, which hinders the acquisition of domain-specific tendencies necessary for effective domain adaptation tasks. “. The general training of a model should not influence its adaptation ability.

3. “These models often misinterpret source domain-specific semantic features as belonging to the target domain, resulting in suboptimal performance for multi-source domain adaptation through prompt learning.” I think prompt tuning is proposed to address the problem that a model use source domain knowledge to complete tasks in target domain.

4. “We train the prompt of the domain with this mixed-up feature.”. what do you mean “the prompt”? what is "the prompt" referring to?

Unclear optimization objectives:

It is unclear what the training objectives of [d] and [v] are. Both t^s and t^k are involved in equations 6 and 7, but it is not clear which prompt is optimized to minimize equation 7.

Unclear annotation:

Figure 3: [v] has been used for interpreting context prompts and should not be used to represent domain features. Furthermore, in equation 8, both f_t and f_s are used to represent features from the target and source domains, and the author should maintain consistency in their terminology.

I may not fully understand the paper since the current version is very difficult to follow and understand. I recommend that the paper undergo careful revision before resubmitting it for peer review.