TransAdapter: Vision Transformer for Feature-Centric Unsupervised Domain Adaptation

1. The paper does not provide sufficient motivation for the proposed components:

• What is the motivation behind introducing the GDD? How does it compare to the conventional domain discriminator, such as the one used in DANN [a]? Please elaborate on the advantages and insights of using GDD in this context.
• Why using double attention module? What’s the motivation and insight? Can the authors explain how this module addresses specific challenges in domain adaptation? Additionally, providing further empirical or theoretical evidence to demonstrate the advantages of double attention over single attention in domain adaptation tasks would enhance the quality of the paper.

2. Most of the comparison methods in Tables 1, 2, and 3 are outdated. It is recommended that the authors include more recent approaches proposed in 2023 and 2024.
3. The experiments on large-scale datasets, such as DomainNet, are missing.
4. The paper's structure can be improved; the content corresponding to Table 4 should be in Section 4.6 instead of Section 5.
5. The formatting of Figures 1, 2, 3, and 4 can be improved. The font sizes in these figures are inconsistent, with some being too large and others too small. Additionally, some elements in these figures require further clarification; for instance, in Figure 3, the origins of ‘V’ and ‘V_shift’ need to be explained.

[a] Ganin et al. Unsupervised domain adaptation by backpropagation. In ICML, 2015.

I have some questions about this paper that need further discussion. Please see them below.

If the authors can address my concerns, I am willing to raise my score.

(a) Presentation and discussions

• Unclear motivation - The fundamental motivation behind proposing improvements to the Swin transformer architecture is not concrete. The authors present the argument of using transformers rather than CNNs for UDA based on the improvements that transformers achieve on several vision tasks. However, they do not discuss the limitations of the Swin transformer and how their method improves on the same.

• Justification of design choices - While the authors have chosen the Swin transformer architecture as the base and provided an intuitive justification of the same, there is no discussion on the design choices behind various aspects of the method. For example, why does the Graph Domain Discriminator (GDD) use graph convolutions rather than regular convolutions? Why use a domain discriminator to align the source and target representations? Why specifically use the third transformer block’s attention output for the GDD? The authors should discuss all these aspects in detail for a good understanding of TransAdapter.

• Discussions on each component of TransAdapter - Even though the authors have provided high-level intuitions behind each component of TransAdapter, this does not convey a clear understanding of the same.

  • For example, in L254-256, the authors mention that integrating the window and shifted window attention mechanism along with reweighting enables the model to capture “multiple aspects of the feature representations”. What aspects does the model capture? How do they help with the UDA task? How are they an improvement over the representations captured by the original Swin transformer architecture?
  • In L223-225, the authors claim that processing the window and shifted window attentions in parallel enables the model to capture long-range dependencies and enhances spatial relationship understanding. There is, however, no discussion on how the Adaptive Double Attention Module achieves any of these aspects. How exactly does the modified attention module capture global dependencies better than a vanilla ViT and a Swin transformer? Moreover, what is long-range dependence in the context of a vision transformer?
  • Overall, the authors should provide detailed discussions on each aspect of the proposed method, and how they improve over the corresponding components in the Swin transformer. The lack of these discussions makes it quite challenging to understand the motivation behind the work.

• Use of terminologies - The authors make use of several terminologies without providing a concrete definition or intuition behind the same. For example, the authors specify “global and local adaptation” without explaining these terms. Do they refer to the alignment of global and local features across the source and target domains? What do global and local features refer to in the context of UDA and the Swin transformer? The authors should clearly specify what these terms mean and how they relate to the context of UDA.

(b) Experiments and results

• Missing comparisons- While the authors have compared with several prior works that utilize the ViT backbone, the comparisons are not extensive and complete. Ideally, the authors should extend the ResNet-based methods to Swin or ViT for a fair comparison. The goal is to ensure a fair comparison by comparing the proposed method with prior works using the same / similar architecture. The list of comparisons missing is given below:
  • SHOT and [R2, R3, R5] with ViT
  • SHOT and [R2, R3, R5] with Swin transformer
  • [R1] with DeiT
  • SSRT [R4] with ViT, DeiT and Swin transformer
• Missing results- The authors have not presented results on DomainNet, which is the most important and challenging UDA benchmark. They should present results on DomainNet, either in the single source or multi-source / multi-target setting (in case of time and compute constraints) to demonstrate the effectiveness of the approach. Additionally, how does the proposed method perform in a multi-source or multi-target setting?
• t-SNE visualidations- There is no discernible difference among the four plots in Fig. 5. Moreover, there are no details provided on the setup for the visualisation experiment. Which layers’ features were considered for visualisation? What is the expected pattern in the plots? What is observed in the plots and what does that indicate about the proposed method? The authors should answer all these questions for a clear analysis of the proposed method with respect to the visualisation.

(c) Missing references

• R1 - Sanyal, Sunandini, et al. "Domain-specificity inducing transformers for source-free domain adaptation."ICCV 2023.
• R2 - Hoyer, Lukas, et al. "MIC: Masked image consistency for context-enhanced domain adaptation." CVPR 2023
• R3 - Yang, Shiqi, Shangling Jui, and Joost van de Weijer. "Attracting and dispersing: A simple approach for source-free domain adaptation." NeurIPS 2022.
• R4 - Sun, Tao, et al. "Safe self-refinement for transformer-based domain adaptation." CVPR 2022.
• R5 - Zhang, Yixin, Zilei Wang, and Weinan He. "Class relationship embedded learning for source-free unsupervised domain adaptation." CVPR 2023.

1. Overall, the three modules in TransAdapter are specifically designed for Swin Transformer, and they appear difficult to generalize to other models, even those based on Transformer like ViT. Additionally, the paper does not provide any experiments or discussions that can be referenced.

2. The motivation for using Swin Transformer is stated in lines 58-62 of the main text. The relevance of long-range dependencies and non-overlapping patches processing to demonstrate that Swin Transformer is suitable for UDA tasks does not seem strong. The authors need to provide more insights into this motivation.

3. The paper states that the "Graph Domain Discriminator (GDD) leverages key features from the third transformer block’s attention mechanisms to capture fine-grained details," as mentioned in line 161. However, there is no discussion in the paper on why the third block is the best. The authors need to provide additional ablation studies to explain this choice.

4. The authors mention the use of Gradient Reversal Layer (GRL) in lines 194-199. However, they do not explain how GRL is applied in GDD, and there is no explanation elsewhere in the paper, not even in the figures.

5. The references to $f$ and $g$ in Equation 6 are unclear. Section 4.4 (Objective function) is inappropriately placed in the experiment section.

6. In Section 4.2, the authors state that CutMix and MixUp are used as data augmentation strategies. However, there is no ablation study on their effects, making it difficult to discern whether the performance improvement of TransAdapter comes from data augmentation.

7. The paper only uses Swin-B for experiments. At least Swin-T, Swin-S, and Swin-L should be used to demonstrate the reusability of TransAdapter across different Swin Transformer models.

8. The paper lacks ablation studies on hyperparameters, such as $\lambda_{local}$ and $\lambda_{global}$.