Robust Core-periphery Constrained Transformer for Domain Adaptation

1. Why the improvement appears to be smaller on DomainNet than other datasets? On DomainNet, RCCT: 46, CDTrans: 45.2, while on Visda, RCCT: 90.7, CDTrans: 88.4. I know DomainNet is more difficult to handle, but we still can have some further discussions.
2. I am also curious about the performance on other domain adaptation settings, such as Partial Domain Adaptation. Can we have a discussion?

1. The novelty of this paper may be insufficient. The "core-periphery" attention is simply a slightly modified standard multi-head self-attention. "Latent feature interaction" seems to be a simple feature-space mix-up. None of them are novel. Their connection to the biological brain is simply far-fetched.

2. All the citation formats in the paper are incorrect. Many equations include errors, e.g., Eqs. (3-4).

3. As shown in Eq. (15), the proposed method introduces too many additional hyper-parameters.

4. As shown in Table 5, the contribution of "core-periphery" may be trivial.

5. Given the aforementioned issues, "a large margin of performance improvements" is not a proper reason to accept this paper.

1. The presentation of the paper is very poor, which made it hard to understand the core contributions

• For a paper which involves modifications of an architecture (ViT), it is crucial to introduce the components of the basic architecture to contextualize the modifications and be self-contained. This paper does not do this.
• There is no proper organization of the methodology section. Sec 3.4 uses a graph constructed in Sec 3.3.1, however, the relevance of the graph is never mentioned in the latter. This leads to a "back and forth" narrative structure.
• The equations and notations are all over the place. Eq. 3 uses the indices h and b for summation, however, the rest of the equation uses some other indices. C is mentioned as a scalar, but somehow the matrix transpose operation is used. Similarly, f_ip is used to dennote the r-th token of i-th image - what is r here?
• Eq. 9,10 includes the V inside the softmax.

There is no clear structure and mathematical notation is not used or explained properly.

2. In the latent-feature interaction (LFI), the operation defined in Eq. 8 does not make sense to me. It involves mixing the features of two distinct images without mixing the labels of the instances. This disregard of the class information is confusing. The authors draw a weak analogy to noise in brain processing, but no proper grounded explanation is given beyond "The LFI operation helps generate robust CP graphs." How?

3. The ablation studies in Table 5 shows that a huge chunk of the performance improvements come from the mutual information loss, which has been used before in multiple UDA papers [a]. The ablation for the global domain discriminator is also not present.

4. As shown in Fig 3, different UDA setups require very different mixture values for the LFI operation. It is not clear how these values are chosen.

5. None of the baselines are discussed in the text or cited. I have no idea what these methods are since they are simply acronyms in the paper.

[a] Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation, ICML 2021

• Important ablation studies are missing

  • There are no ablation studies on the different loss terms involved. Hence, it is unclear which losses contribute more or less to the performance. For example, is the global discriminator required if the patch discriminator is already performing a similar task?
  • In Table 5, CP and LFI are added to a baseline that already uses SCM (and SCM gives the most improvement compared to CP and LFI). But we also need to check how well CP and LFI can work on their own since the motivation for these components does not mention the necessity of SCM.
  • Hence, it seems that the self-clustering loss (that is used by many different works) contributes the most while adversarial training with discriminators is less effective (and more complicated to optimize). This potentially weakens the claims of how important the core periphery constraints are.

• Regarding novelty of LFI

  • It seems very similar to mixup at the feature-level (for example in [W1]). In general, mixup is also a fairly common training augmentation. Please compare the similarities and differences since it hurts the technical novelty of LFI.

• Overall framework seems to be complicated

  • Since the method involves adversarial training with a global discriminator and a patch-wise discriminator, the method might be difficult to tune or optimize. Further, there are no hyperparameter sensitivity analyses for the loss weighting hyperparameters like $\alpha, \beta, \gamma$.
  • As per the Supplementary, different hyperparameters are used for different benchmarks, so it is unclear how much tuning is required for a new benchmark.

• Some important details of the approach are unclear

  • Page 5: there is no $r$ in the equation for $C(f\_{ip})$, so it is unclear to me.
  • Eq. 3: $h$ and $b$ are not involved in the inner terms and the dimensions of $C(f\_{ip})$ are unclear. From the original definition, $C(f\_{ip})$ seems to be a scalar but a vector inner product is being computed in Eq. 3.
  • Eq. 4: what is $M(i, j)$? How is it computed? Is it actually Eq. 3 which defines it as $M\_{cp}$?
  • Eq. 4: why is the threshold chosen to be 0.5? How sensitive is the approach to changes in this threshold?
  • Sec. 3.3.2: what is the meaning of “sequentially chosen”?

References

[W1] Lim et al., “Noisy Feature Mixup”, ICLR22